wa 


‘ 


WH 


ot 


O 


Offset 


) 


? 


J GUIBERT 


er 


Rae 


> 
rv, 


— 


Donated by Dr. V Benjamin on 24/12/2004 


Educational 
Handbook for 
Health Personnel 


Sixth Edition 


J.-J. Guilbert 


WHO Offset Publication No. 35 


WHO offset publications are intended to make generally available material that for economic, 
technical, or other reasons cannot be included in WHO's regular publications programme and 
would otherwise receive only limited distribution. They are usually reproduced by photo-offset 
from typescript, rather than by letterpress, and do not necessarily receive editorial revision. 


a Ce livre est publié en francais par l’OMS sous le titre: Guide pedagogique pour les personnels de santé. 


Fs] Information concerning Arabic, Bulgarian, Czech, Farsi, German, Hungarian, Indonesien, Italian, Polish, 
Portuguese, Russian, Serbo-Croat, and Spanish editions is available from WHO, Geneva (attention EPM/HMD). 


ISBN 92 4 170635 X 
ISSN 0303-7878 
©World Health Organization 1987 


Reproduction and translation authorized, for nonprofit educational activities provided that WHO Origin is 
mentioned and that copies containing reproduced material are sent to: Chief, Office of Publications, World 
Health Organization, 1211 Geneva 27, Switzerland. : 
TS lal ee and fell Presentation of the material in this publication do not imply the expression 
lon whatsoever on the part of the Secretariat of the World Heal izati 

th Organization c i h 

status of any country, territory, city or i iti . ; wma ts hele 
; ; area or inc imitati i i 

“hele of its authorities, or concerning the delimitation of its frontiers 


The author alone Is responsible for the views expressed in this publication. 


Printed in Switzerland by Imprimeries Populaires, Geneva 
87/7136 — 4000 (R) 


PRINTED IN SWITZERLAND 
84/6261 — Populaires — 2500 (R) 


Educational 
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Health Personnel 


How to help educators to increase their skills so 
as to make learning easier for the students 


1987 


objectives of the handbook 


When you have studied this handbook you should be able to: 


define the professional tasks which will provide the basis for determining educational 
objectives | 


plan a curriculum 


construct tests and other measurement devices. 


These skills will be based upon a measurable gain of knowledge concerning: 


relationship between professional training programmes and subsequent practice; 
principles of learning and role of the teacher; 

role of objectives in educational planning; 

principles and methods of curriculum planning; 


principles and practice of educational evaluation. 


You will also have strengthened your desire to go on learning and acquiring skills in 
education and will have mastered an effective way of setting about it. 


Contents 


Objectives of the Handbook 

For whom is this Educational Handbook intended? 

How to use the Handbook 

Identification of your needs as an educator 

Pre-test 

List of educational objectives 

Theoretical background that will help you reach the educational objectives 
Table of exercises proposed in the Handbook 
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Chapter 1 Educational objectives 


The educational spiral 

Importance of defining professional tasks 

Selection of training goals 

Example of services provided by rural health units 
Types of educational objectives 

General objectives: professional functions 
Professional activities and intermediate objectives 
Built-in relevance approach to educational planning 
Professional tasks and specific educational objectives 
Identifying the components of a task 

Construction and critique of specific educational objectives 


Chapter 2 Evaluation planning 


What is evaluation? 

Formative and certifying evaluation 

Aims of student evaluation 

Selection of an evaluation method 

Advantages and disadvantages of different types of test 

Qualities of a measuring instrument (validity, objectivity and practicability) 
How to organise an evaluation system 


Chapter 3 Programme construction 


Distinctive attributes of education for the professions 
Curriculum planning principles 

The purpose of teaching is to facilitate learning 
Teaching: a complex but challenging task 

Teaching methods and educational media 
Self-learning packages 

The concept of integrated teaching 

The concept of integrated learning 

Planning of programme reform 

The use of specification tables 


Chapter 4 Test and measurement techniques 


Why evaluate? 
Guidelines for evaluating a training programme 
Points to consider in evaluating programme changes 
Guidelines for evaluating educational objectives 
Programme evaluation by the students 
Evaluation of students’ level of performance 
Assessment of practical skills 
Assessing attitudes 
Written tests 
The programmed examination 
Stages of assessment 
Test construction specification table 
Relative and absolute criteria tests 
Item analysis 


Chapter 5 How to organise an educational workshop 
Chapter 6 Index and glossary 


Chapter 7 Bibliography 


preface to the sixth edition 


This sixth edition (1987) brings up to date, 
and improves on, the previous edition. It is 
a collective work taking into account the 
critical comments and suggestions of 
numerous users of the earlier editions and 
the results of a survey carried out among 
200 selected readers, whom we take this 
opportunity of thanking. 


The first edition was prepared on the basis of 
documents distributed during a workshop 
organized by the Regional Office for Africa, 
Brazzaville, in December 1969 with the 
assistance of consultants in medical 
education.! It has been widely used in the 
Region and served as a basic document for 
many educational planning workshops. 


It then became evident that it would be useful 
to give the reader better guidance during his 
progression through the whole educational 
planning cycle. Although this objective would 
most likely be achieved by a completely 
programmed presentation it seemed that the 
complexity of the problems involved was not 
great enough to justify a method which would 
be so intricate and long to prepare. Only 
practice could say whether this was correct. 
In 1975 a fully revised text was field tested 
and led to the 1976 revision distributed in over 
10,000 copies in four languages. The 1981 
edition was translated into 14 languages as a 


result of national initiatives. It too was used 
at many “educational workshops”, during 
which interesting and constructive criticism 
was expressed to the same effect as the 
comments received from the readers partici- 
pating in the survey, who included many 
experienced educators: need for greater 
clarity (less jargon), more concrete examples 
and more practical exercises. 


Another attempt has been made in this edition 
to take this advice into account. 


Like the previous ones, this new edition will 
be revised in its turn, in the light of the 
suggestions we hope to receive from users. To 
order copies please see the list of addresses at 
the back of the book. For editions in other 
languages, or if you would like further 
information or explanations, do not hesitate 
to write to: 


J.-J. Guilbert, M.D., Ph.D. (Educ.) 

Chief, Educational Planning and Methodology 
Division of Health Manpower Development 
World Health Organization 

1211 Geneva 27 

Switzerland 


Your suggestions will always be welcome. 
There is a special sheet you can use for 


them at the end of this handbook. 


1 Dr. S. Abrahamson and Dr. H. Peterson (USC, Dept. Med. 
Educ., Los Angeles). 
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Dear Colleagues, 


In our common struggle to achieve the objective of ‘‘Health for all by the year 2000”, the place 
of basic training and continuous education of health personnel is a crucial one. 


The Conference of Alma-Ata recommended, among other things, that governments “ [define] 
the technical role, supportive skills, and attitudes required for every category of health worker 
according to the functions that need to be carried out to ensure effective primary health care...”’. 
It also recommended that “health workers, especially physicians and nurses, should be socially 
and technically trained and motivated to serve the community ; that all training should include 
field activities”, and that due attention should be paid to the preparation of teachers of health 
workers. 


All these reasons lead me to regard the publication of this revised edition of the Educational 
Handbook for Health Personnel as particularly useful, in that it will facilitate the application 


of those recommendations. 


It is also a practical example of the technical support that an international organization such as 
ours must offer to back up national efforts in favour of primary health care. 


Of course it is only one element among many; a small stone that helps pave the difficult way to 
health for all. 


I hope this Handbook will continue to meet with the succes it has enjoyed for over 15 years 
among its many users. 


Dr. H. Mahler 


Director-General 


1211 GENEVA 27-SWITZERLAND Telegr.: UNISANTE-GENEVA Telex: 27821 OMS 1211 GENEVE 27-SUISSE Télégr.: UNISANTE-GENEVE 


for whom 


is this educational handbook intended? 


Originally, in 1969, the first edition was 
written for teachers of the health sciences. 
Subsequently, however, the Handbook was 
used above all by hundreds of participants in 
meetings organized by WHO or held with WHO 
assistance (educational planning workshops, 
teaching methodology seminars, etc....... ). 


A survey of 200 users carried out in 1975 
revealed the following opinions. 


A very small minority (10%) felt that the 
Handbook should be supplied only to partici- 
pants in controlled educational activities 
(workshops, seminars, courses, etc.) or 
reserved for ‘‘teachers of teachers”. The 
majority, however, felt that dissemination 
should be as wide as possible, and be directed 
to all teachers of health sciences at all levels, 
to health administrators with staff supervision 
responsibilities, and to students, so as to help 
them to draw maximum benefit from their 
learning activities and participate in their 


organization. The latest poll, conducted in 
1978, produced very apposite comments 
from about 100 users in all parts of the world 
and the consensus is still in favour of the 
widest possible distribution. Many teachers of 
the various health professions (dentists, nurses, 
sanitary engineers, physicians, pharmacists, 
etc.) have stated that the Handbook answers 
their needs, but some think that the author’s 
training as a physician has still too often 
biased the choice of examples. A further 
effort has been made with this 1981 edition 
to produce a text better suited to everyone 
working in the health professions. It is, how- 
ever, very difficult to strike a perfect balance, 
for many reasons. While it is still true that 
most of the examples relate to the medical 
and nursing professions, it is the author’s 
hope that each user of the Handbook will 
make a personal effort at adaptation , replacing 
the examples given by other more suitable 
ones whenever necessary. 


jargon and meaning of words 


Many users of the previous editions have 
asked for special care to be taken to simplify 
the language used. We hope they will be 
satisfied. It has not, however, been possible 
to avoid using certain technical terms (for 
example: discrimination index). Neither has 
it been possible to avoid assigning precise and 
restricted meanings to words which are often 
used interchangeably in everyday parlance 
(for example: task, activity and function). In 
all such cases the words are defined in the 


Glossary (p.6.01 et. seq.). 


It is very important that we should understand 
one another, and for that we must give 
identical meanings to the words we use. But 
it is just as important not to get bogged down 
in endless discussions. Your aim is not to 


draft definitions of words for a dictionary. 


So please accept the definitions proposed in 
this Handbook, at least while you are using 
1t; 


how to use the handbook 


The presentation and layout of this Handbook 
are unusual. 


# It should be stressed at this point that the 
approaches suggested in this book are the 
result of a deliberate choice by the author 
and reflect his ideas in the field of education. 
Various theories sometimes regarded as con- 
tradictory are explained to the reader so as 
to give him food for reflection, rather than 
to subject him to any philosophical con- 
straint. It is for the reader to make his 
choice, to draw initial conclusions and, in 
particular, to seek solutions for his own 
teaching problems by taking what is worth 
using among the various theoretical 
approaches suggested. To reject them whole- 
sale would hardly be constructive; it would 
be preferable to propose better ones. 


# For each chapter a list of objectives is 
included (yellow pages) to give you an 
idea of what you will get out of the chapter. 


The theoretical input (white pages) is 
presented in the form of original docu- 
ments or short quotations from texts 
listed in the bibliography. 


All through the Handbook there are 
exercises (blue pages) to help you to 
determine whether you have increased 
your skills (see check list, page 15). 


Certain documents are printed in large 
type so that they can be easily made into 
transparencies for use with an overhead 
projector (see technique on page 3.45). 


= The page numbering is also unorthodox: 
every chapter begins with a number ending 
in 01 (for example page 3.01 is the first 
page of the third chapter), however many 
pages there may be in the previous chapter. 
This meakes it easier to find a specific 
passage in the Handbook. 


Rather than try to explain the whys and 
wherefores of this unusual layout, which 


some find irritating and complicated at 
first, and others as intriguing as a detective 
story, it seems simpler to invite you to use 
this book just as it is. You will then draw 
your own conclusions as to the “reasons’’ 
for its layout. There must be some... even 
if the author is not aware of them all! 


In any case you are warned not to read this 
Handbook like a conventional book, starting 
with the first page and hoping to get to the 
last. It is meant to be used as determined 
by the questions you put to yourself, your 
needs, and the teaching problems that 
bother you — by your own objectives, in 
short, whether or not they are already 
articulated. 


This is why we are going to try and help you 
in this essential but unusual and therefore 
difficult undertaking. 


First situation: you are alone (on a desert 
island) or else you are accustomed and prefer 
to work alone. You are going to need grit, 
perseverance and 35 to 50 hours of free 
time. 


Good Luck! 


Second situation: a colleague also has the 
Handbook and agrees to work with you. 
It will not be such an uphill task. If you 
can get together a small working group of 
five persons, still better. 


Third situation: you are taking part in an 
educational workshop. The working 
methods for such a meeting are described 
in this Handbook (pp. 5.10 — 5.12) and 
group dynamics will do the rest. 


In any case, the first thing to do is to iden- 
tify your needs. 


You are invited to proceed in three stages. 


‘dentification of your needs as an 
educator 


First Stage 


= What are the educational ‘questions’ that nurses” or ‘teaching how to interview a 
you would like to have answered? (For patient’). 
example: “what is an educational object- 
b 9? «66 ¢ piel ? 
ive? Ae does the word pees Bat = What are the educational “problems” you 
with reference to an examination?”’). would like to solve? (For example: “I am 
= What are the educational ‘‘tasks’’ that you in charge of 600 students in different 
would like to take on? (For example: university years: how am I to organize field 
“organizing a nutrition course for student work in preventive medicine?” 
NOWerosto-% Try to draw up a preliminary list of your “questions”, ‘“‘educational tasks” or 


“problems” for the coming 12-month period. 


identification of your needs as an 


educator 


Second stage 


Would you like to find out how much you 
know before studying the rest of the Hand- 
book? One way of doing this is to try to 
answer the questions in a pre-test*. 


To do so, read straight away the questions on 
pages 1.57 — 1.60, 2.45 — 2.47, 3.79 — 3.83 
and 4.75 — 4.80 and enter your answers on 
the answer sheet, page 10. 


if your score is low in this pre-test, this should 
be a source of satisfaction, for it may show 
that you were right to start perusing this Hand- 
book and that it will be worth your while 
continuing so that you can improve your 
weak spots. 


The test will help you make a personal diag- 
nosis of the teaching areas you need to bring 


up to date. Your aim in doing this test is not 
to find the right answer but simply to take 
note of the fact that you do not find it! 


You may also rest assured that your score in 
the post-test (page 5.34) will be another occa- 
sion for satisfaction for it will show an appre- 
ciable gain after you have used the Handbook. 


If your score in the pre-test is high, you should 
choose areas which are still ‘“‘uncertain”’ (those 
for which you did not find the correct answer) 
and go into the matter more thoroughly. 


*This pretest deals only with theoretical knowledge relating 
to group 2, page 2. You will have an opportunity of gauging 
how far you have achieved the group 1 objectives by doing 
other exercises which appear throughout the Handbook 
(blue pages). 
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identification of your needs as an 


educator 
Third stage 


To use this Handbook effectively for your own 
needs, it is proposed that you choose from 
among the following educational tasks or 
objectives (broken down into four main 
themes) the ones which interest you and in 
which you would like to achieve a degree of 
competence. 


The time required for personal work (study 
of the text, practical exercises) on some of 
these tasks will be more or less long. Some 
call for a few minutes’ work, others several 
hours to allow for study of the documents 
suggested (p.14). 


To guide you in this first choice, do not 
forget to take into account the ‘‘questions 
and problems” you listed on p.8 and your 
weak and strong points (results of pre-test, 
p. 5.34). 


Limit this initial choice to about 10 tasks, 
circling the numbers of those that seem 
most important for you on the list (numbered 
1 — 40) appearing on pages 12 — 13. 


It is natural that you should have some diffi- 
culty in making your choice at this stage, for 
you are not yet familiar with ‘‘educators’ 
jargon”. Do not forget the Glossary (pp.6.01 
et seq.) Don’t worry ......... in any case there 
is no risk involved! 


Moreover, you can always adjust your choice 
as your work progresses. 


Once your initial choice has been made, turn 
to p. 14 and follow the instructions given 
there: study the suggested pages and do the 
corresponding exercises (blue pages.) 
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educational 
objectives 


Define the following terms: professional 
task, activities, functions, role, institut- 
ional objective; specific objective; 
domains of practical skills; communic- 
ation skills and intellectual skills. 


List the qualities of an educational 
objective and the sources necessary to 
ensure its relevance. 


Define the professional functions of a 
member of the health team whom your 
teaching institution is responsible for 
training (general educational objectives) 
so as to deal with the health problems of 
society.” 


Analyse a major professional function by 
defining the various intermediate com- 
ponents (activities) making it up.* 


Define a professional task and identify 
its components (domains of practical 
skills, communication — skills and 
intellectual skills). 


Draw up a list of the specific educational 
objectives relating to a professional task, 
stating explicitly what you feel the 
student should be able to “do” after a 
given course of instruction (that he 
was not able to do previously) and 
corresponding to the “enti of the 
communication skills or practical skills 
involved in this task.* 


Taking a specific objective in a non- 
intellectual domain (i.e. practical or 
communication skills), define in the 
form of specific educational objectives 
what theoretical knowledge you feel 
the student should possess if he is to 
attain that objective.” 


Make a critical analysis of specific 
educational objectives (listed by a 
colleague), indicating in particular 
whether they include all the requisite 


elements (act, content, condition, 
criteria). * 


Draw up a list of the possible reactions 
of colleagues in your faculty to the idea 
of having to define educational 
objectives derived from professional 
tasks and propose strategies for over- 
coming those reactions, * 
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evaluation 


14. 


1x? 


te: 


14. 


153 


16. 


dys 


a; 


planning 


Draw a diagram showing the relation- 
ship between evaluation and the other 
parts of the educational process. 


Define the principal role of evaluation, 
its purpose and its aims. 


Describe the difference between forma- 
tive and certifying evaluation. 


List the good and bad features of a test. 


Compare the advantages and disadvan- 
tages of tests in current use. 


Define the following terms: validity, 
reliability, objectivity, and describe the 
relationship that exists between them. 


Choose an appropriate evaluation 
method (questionnaire, written examin- 
ation, “objective” test (MCQ or short, 
open-answer question) or essay question, 
oral examination, direct observation, 
etc.) for measuring the students’ attain- 
ment of a specific educational objective. 
Compare the alternatives in a specific- 
ation table.* 


Define (in the form of an organizational 
diagram) the organization of an evalu- 
ation system suitable for your establish- 
ment, and list the stages involved. 


Indicate: 


(a) the most important educational 
decisions you have to take; 


(b) the data to be collected to provide 
a basis for those decisions; 


(c) the aims of the system and sub- 
systems in terms of decisions to be 
taken and the object of each 
decision (teachers, students, pro- 
grammes). * 


Identify obstacles to and strategies for 
improvement of a system of evaluating 
students, teachers and programmes. * 


* See footnote, p. 5.19. 


19. 


20. 


Pay 


2. 


24. 


ees 


26. 


Ae 


28. 


29. 


programme 
construction 


Explain the differences between 
dedication “teaching” and “‘learning”’, 
and describe the new trends in the 
teaching/learning system and the various 
learning situations. 


Define the concept of relevance of a 
programme, 


List 10 conditions which facilitate 
learning on the basis of the list and evaluate 
a specific learning activity. 


Indicate the aims and general methods of 
teaching. 


Specify at least two acy tapes and two 
disadvantages for each technique and 
medium used in teaching. 


method that will make 
objective easier to 


Seiect a teachin 
an educationa 
achieve. 


Compare the alternatives in a specific- 
ation table.* 


Indicate the steps involved in con- 


structing a programme. 


Construct a programme or decide 
whether a programme or course needs 
revision, using a specification table. * 


Construct a self-learning package. * 


Define the role which, as a teacher, you 
would like to assume in order to motivate 
and facilitate the learning of students 
for whom you are responsible.* 


Identify the obstacles liable to be 
encountered in setting up a competency 
based curriculum geared to the health 
needs of the community and describe 
strategies for overcoming them. * 


implementation 


30. 


= 


Bs 


33. 


34. 


a3. 


36. 


37. 


38. 


39. 


40. 


of evaluation 


Indicate the different elements that 
should be considered in the evaluation of 
a teaching programme. 


Indicate the different elements that 
should be considered in the evaluation of 
the educational objectives of a teaching 
programme. 


Define the advantages and limitations of 
a system of evaluation of teaching by 
the students. * 


Construct an observational rating scale 
and/or a practical test to evaluate the 
behaviour of a student in the domain of 
communication and/or practical skills.* 


Propose a question for a written (open- 
book) examination of the “essay” type 
Or a series of six short, open-answer 
questions and indicate the norms of 
performance permitting — objective 
marking (marking table).* 


Draw up three multiple choice questions 
(MCQ) in the domain of intellectual 
skills — at least two of the objectives 
must measure an intellectual process 
superior to level 1 “simple recall’’ (either 
level 2 “interpretation of data” or level 
3 “problem-solving’’).* 


Indicate the advantages and limitations 
of a programmed examination. * 


Define the following terms: prerequisite 
level test, pre-test, interval test, com- 
prehensive pre-final; indicate their pur- 
pose and the stages at which they are set. 


Explain the difference between a relative 
and an absolute criteria test. 


Calculate the acceptable pass level for a 
MCQ examination and establish the 
scoring criteria and norms which permit 
determination of the passing grade of a 
mini-test (made up of the questions 
mentioned in objectives 34 and 35).* 


Do an item analysis of a question (cal- 
culate the difficulty index and the dis- 
crimination index) and draw the relevant 
conclusions. 
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Theoretical background that will help you 
reach the educational objectives of the workshop 


Study the following 


Study the following 
pages of the 
Educational Handbook 


pages of the 


For objective 
Educational Handbook 


For objective 


Glossary and pp. 1.05-1.07, 
He1.7: dnd Sal elo oy 


1.36 
Beil 2Uabedsetla toast) 
1.48-1.50 


pp. 3.28, 3.29 
pp. 3.21, 3.22 


Obj. 1 and pp. 1.09-1.27 pp. 3.41-3.47 


Obj. 16, Obj. 23 and pp. 3.22, 


pp 3.03 —3.49 (*) 


pp. 1.17-1.19, 1.33-1.41 


Obj. 25 and pp. 3.59-3.75 or 


Obj. 2, Obj. 5 and pp. 1.43 
pp. 1.2753.73, 40+ 


aldose Le 3.0 


Ob ound pp. ko 41.56 Obj. 24 and pp. 3.49-3.54 


Obj. 6 and pp. 1.56, 
4.10-4.14 


Obj. 1-andipp.1,63.-3.71, 
379 


Obj. 20; Obj. 27 
and pp. 3.33-3.40 


Obj. 28 and/or Obj. 26 and 
pp. 3./i,'9.1o- eu 


pp. 2.03-2.05 pp. 4.03-4.09 


pp. 2.02-2.14, 2.19 pp. 4.10-4.14 


Dope. -2.5 pp. 4.15-4.19 


pp. 2.26-2.29 Obj. 16 and pp. 4.22-4.27 aa) 


pp. 2.30 and 2.31 Obj. 16 and pp. 4.28-4.30 


pp.2.33=2.37 Obj. 16 and pp. 4.31-4.40 
Obj. 6 and pp. 2.21-2.23, 


2227-2538 pp: 441-448 


Obj. 11 and pp. 2.40-2.43 pp. 4.49-4.53 


Obj. 17 and pp. 2.44, 3.71 
375 p. 4.61 


pp. 1.04, 3.18-3.29 ri eee Obj. 38 and pp. 4.62, 


pp. 1.33, 3.04-3.12 pp. 4.65-4.73 


_ and do the corresponding exercises (blue pages): see list on next page. 
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3 Identifying professional activities 


Page 
by = 


3 Defining the main functions of health personnel 1.21/1.22 


20/26 | Analysis of the relevance of a programme 


5 Identifying components of a professional task 


—" 
x] 


Selecting active verbs relating to a task 


Statement of educational decisions 


12 Distinguishing between formative evaluation and certifying evaluation 


13,14,16| Selecting a method of evaluation | 
r 
18 Identifying obstacles to and strategies for applying an evaluation system 
9 
28 
z 
9 
9 
2 
3 
33 Constructing an attitude table 
Z 
z 
35 Preparing multiple-choice questions (MCQ) 
: 
6 
9 


40 Calculating the difficulty index and discrimination index for a 
question 


30/40 | Evaluation of knowledge about test and measurement techniques 
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Identifying the elements of an educational objective 
Evaluating an educational objective 


ee ee 
miu} ap] op 
wpe 1 AX] 


_— 
ws 
TS 


aay 


ok 
wi ws 
NN} wm 


1.57/1.60 
2.09 
ZAG2.12 
2a2 3 
238 
2.43 
2.44 
2.457297 
ae 
3139: 
Me hg 
3.58 
3.77 
3.78 
3479/3183 
4.19 
4.22 
4.25 
4.29 
4.30 
4.39/4.40 
4.48 
4.55/4.57 
4.71 
4.75 


4.75/4.80 


1 


Vi 


acknowledgments 


This revised edition of the Handbook has 
‘nvolved a lot of meticulous and time-con- 
suming work over the last three years. Many 
people have offered advice and help and have 
supplied documents used in the text. 


unfailing interest and has carefully read and re- 
read the intermediate drafts. 


We wish to express special thanks to: 


S. Abrahamson (Los Angeles), C. McGuire and 
G. E. Miller (Chicago) for their appreciative 
comments and even more for their useful and 
judicious advice. 


From the start, the encouragement, support, 
advice and patience of T. Fulop, Director of 
the Division of Health Manpower Develop- 


ment, have been invaluable. We should also like to thank all those who have 


B. Pissaro (Paris) has followed our work with sent us helpful comments and suggestions: 


B. Adjou-Moumouni (Lome), Z.S. Al-Alway (Dacca), P. Alexander (Minneapolis), —L. Allal 
(Geneva), J. L. Argellies (Tunis), W. Barton (Geneva), G. R. Beaton J ohannesburg), A. Benadouda 
(Algiers), D. Benbouzid, (Algiers), J. Beneke (Copenhagen), D. Benor (Beer Sheva), V. Bergk 
(Heidelberg), P. J. Blizard (New Delhi), C. Boelen (Tunis), J. Brines (Valencia), S. Bukkavesa 
(Bangkok), L. Burke (Mogadiscio), F. Canonne (Paris), P. Carteret (Lome), P. Chaulet (Algiers), 
D. Clark (London), A. Coelho (Lisbon), A. L. Courtieu (Nantes), K. R. Cox Gydney),T. O. Crisp 
(Dacca), L. Daufi (Barcelona), R. Debre (Paris), H. Dieuzede (UNESCO), J. F. D’Ivernois (Paris), 
C. Di Pasquale (Dakar), S. R. Dodu (Accra), M. A. C. Dowling (Geneva), C. Engel (Newcastle), 
V. Ermakov (Moscow), W. Felton-Ross (Bloomfield), J. -J. Ferland (Quebec), D. Flahault 
(Geneva), H.H. Freihofer (Bern), A. T. Garcia (Aden), S.Goriup (Geneva), J. P. Grangaud 
(Algiers), J.P.Grillat (Nancy), V.F. Gruber (Basel), R.M.Harden (Dundee), E. Harper 
(Allahabad), H. D. Houran (Kuwait), C. landolo (Rome), T. Jaoude (Mogadiscio), C. Jaramillo 
Trujillo), H. Jason (Miami), T. Javor (Pecs), O. Jeanneret (Geneva), M. Jegathesan 
(Kuala Lumpur), S. Jones (Visakhapatnam), J. Kadish (Washington), I. Kapoor (Bombay), 
F. Katz (Geneva), J. Kemp (San Jose), H. Kolb (Vienna), X. Kondakis (Patra), S. R. Kottegoda 
(Colombo), M. Ladjali (Algiers), G. Lavoipierre (Geneva), T. B. Lee (Kuala Lumpur), M. Lenghi 
(Benghazi), G. Lotti (San Remo), I. M. Lovedee (Rangoon), E. Mabry (Bangalore), 
V.G. MacDonald (Wellington), N.McIntyre (London), J.M.McLachlan (Kuwait), S. Maes 
(Antwerp), A. Maglacas (Geneva), M. Manciaux (Paris), W.F.Maramis (Surabaya), A. Mazer 
(Marseilles), D. Melliére (Paris), A.Mejia (Geneva), P.I.Mintchev (Sofia),  S. Mokabel 
(Alexandria), R.Mol (Veldhoven), K.Mowla (Islamabad), E.Muret (Paris, M. Nichol 
(Katmandu), G. Nicolas (Nantes), C.Nihoul (Ghent), N.M. O’Brien (Amman), G. H. Pauli 
(Bern), M. Pechevis (Paris), P.Pene (Marseilles), S.L.Purwanto (Djakarta), C. A. Quenum 
fone! A.A. Rahman (Jitra), A. Rodriguez-Torres (Valladolid), J. E. Rohde (New York), 
e Ne (Geneva), A. M. Sadek (Alexandria), J.C. Salomon (Paris), G. Scharf (Woipy), C.Searle 
en Hg: eBRete (Bangkok), S. Slimane-Taleb (Algiers), A. Stewart (Dundee), 
Vie (Calcutta), A. E. Suliman (Dacca), G.Szabo (Debrecen), A. Thein (Rangoon), 

. Tigyi (Pecs), A. Torrado da Silva (Coimbra), E. Varga (Debrecen), F. Vilardell (Barcelona), 


K. Vuylsteek (Ghent), J. Vysohlid (Pra 
ap gue) J. S. Wasyluk (Warsaw), W. W 
M. Yamamoto (Tokyo), J. Yip (Djakarta). 4 see a aa 


We trust that anyone who has been overlooked 


Finall 
will forgive us. Indeed, among the 1250 collea- inally, our thanks go to: 


gues who have taken part in workshops with 
the author, there are many whose questions 
and opinions have helped in our efforts to 
ensure that the Handbook is tailored to meet 
the needs of its users. These contributors are 
too numerous to be mentioned by name. 


16 


P. Duchesne, P. Etienne,M. Magnin,E. Riccard, 
M. Schaltegger and M. Wolff and to the 
English editors R. Binz, K.Grinling and 
C. Stevenson without whose untiring work 
this document would never have been com- 
pleted. 


€ 


educational 
objectives 


the concept of educational objectives 


The aims of this first chapter are to 
show that if precision and clarity 0 
problems is even more so; and to sho 
health personnel are better prepare 
problems of society. 


Those in 


= Preparing instructional objectives (1962). 


m Goal analysis (1972). 


show the advantages of defining educational objectives; to 
f educational objectives are important, relevance to health 
w that an approach based on objectives will ensure that 
d to perform professional tasks corresponding to the health 


terested in this approach should read the following works by R.F. Mager. 


= Measuring instructional intent (1973) (Chapter III, pages 15 to 46) Fearon Publishers, 


California, U.S.A. 


And the following publication by the World Health Organization. 


= Criteria for the evaluation of objectives in the education of health personnel, 
WHO, Technical Report Series, 1977, No. 608. 


After having studied this chapter and the reference documents mentioned you should be able to: 


1. Define the following terms: professional 
task, activities, functions, role, institutional 
objectives; specific objective; domains of 
practical skills, communication skills and 
intellectual skills. 


2. List the qualities of an educational 
objective and the sources necessary to ensure 
its relevance. 


3. Define the professional functions of a 
member of the health team whom your 
teaching institution is responsible for training 
(general educational objectives) so as to deal 
with the health problems of society. 


4. Analyse a major professional function by 
defining the various intermediate components 
(activities) making it up. 


5 - Define a professional task and identify 
its components (domains of practical skills, 
communication skills and intellectual skills). 


6. __ Draw up a list of the specific educational 
objectives relating to a professional task, 
stating explicitly what you feel the student 
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should be able to ‘“‘do” after a given course 
of instruction (that he was not able to do 
previously) and corresponding to the domains 
of the communication skills or practical skills 
involved in this activity. 


7. Taking a specific objective in a non- 
cognitive domain (i.e., practical or 
communicative skills), define in terms of 
specific educational objectives what 
theoretical knowledge you feel the student 
should possess if he is to attain that objective. 


8. Make a critical analysis of specific 
educational objectives (listed by a colleague), 
indicating in particular whether they include 
all the requisite elements (act, content, 
condition, criteria). 


9. Draw up a list of the possible reactions 
of colleagues with whom you work in your 
faculty to the idea of having to define 
educational objectives derived from _pro- 
fessional tasks and propose strategies for 
overcoming those reactions. 


eee 
«2 SS eo 


If you are not certain 
|Of where you are 
going 

you may very well 
end up 

Somewhere else 


(and not even know it) 


Mager 
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Every individual should have access to a type of education that permits 
maximum development of his potential and capabilities. 


Education is a process, the chief goal of which is to bring about change 
in human behaviour 


The result of education is 


an expected change in the behaviour of the student in the course of 
a given period. 


The Educational Spiral 


= This ‘behaviour’ will be defined explicitly in the form of educational a 
objectives derived from professional tasks. 


m An evaluation system will be planned so that better educational decisions 
can be taken. 


A programme will be prepared and implemented to facilitate attainment 
of educational objectives by the students. 


The evaluation process will be used to measure the extent to which the 
objectives have been achieved . . . it will measure the student's fina/ abilities 
...and the effectiveness of programme and teachers. 
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the educational spiral 


Programme reform has been a source of 
concern for many years to those training 
health personnel and the alarm has often been 
sounded. However, the strength of the 
traditions impeding necessary reforms has been 
such that it has not been possible to avoid 
serious disturbance in many _ universities 
throughout the world, always caused by a 
reaction in face of the apparent diehard 
conservatism of the system. 


It would, however, be negative and dangerous 
merely to accuse of incompetence those at 
present in positions of teaching responsibility. 
They should be offered help. 


Societies change and have always been 
changing, but until the present century their 
evolution was relatively slow and adaptation 
to change was possible without unduly 
violent disorders. 


The form of teaching has remained unchanged 
for centuries. The university has wrapped 
itself in its privileges and remained deaf to 
the cry from without. The needs of society, 
the practical side of the matter, have been 
left to chance, whereas specific features of 
the situation in each country are changing ever 
more rapidly. Hitherto, unfortunately, little 
or no account has been taken of those features 
and the training of health personnel has 
followed traditional systems. What is required 
now is to make sure that educational pro- 
grammes are relevant. 


There can be no question of continuing to 
copy the models of the past or, in the case of 
developing countries, foreign models. 


The educational system leading to the develop- 
ment of health personnel, at all levels, must 
be re-examined within the context of the 
needs of the country concerned.’ 


No educational system can be effective unless 
its purposes are clearly defined. The members 
of the health team must be trained specifically 
for the tasks they will have to perform, taking 
into account the circumstances under which 
they will work. 


These tasks can only be defined in accordance 
with a plan in which the nature of the services 
to be provided is specified, priorities are 


allotted, the staff needed to provide these 
services determined, etc. .... 


Professional training programmes must then 
be tailored to meet these needs. 


There is room for some degree of optimism 
in this sphere, for no financial assistance is 
needed for a move in the right direction. 
All that is needed is a resource distributed 
more or less equally around the world: mental 
ability. The management of that resource is 
the art of organizing talent and of coping 
intelligently with change. 


Defining the professional tasks of health 
personnel to be trained, the very basis of the 
educational objectives of training centres, 
is of crucial importance. 


Thus an educational programme, instead of 
being the result of a non-selective accumul- 
ation of knowledge built up over the centuries, 
must be shaped selectively in terms of the 
goal to be achieved. If that goal is modified 
in the course of time, the programme must 
also be modified accordingly. 


Definition of professional tasks must proceed 
from a study of needs, take account of 
available resources and indicate clearly and 
precisely what various categories of personnel 
will be called upon to do during their pro- 
fessional careers in a given type of health 
service. 


| The study of needs, organization of health services, and definition of tasks and functions are, however, not dealt with in this Handbook. 
Consequently, specialized texts should be consulted concerning those aspects (see Bibliography, p. 7.01). 
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The object of education is not to shape citizens to the 
uses of society, but to produce citizens able to shape 


a better society. 
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the educational spiral 


Defining 
Tasks and 
Educational Objectives 


Implementing 


Evaluation 


Preparing and 


Implementing an 
Educational Programme 
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Planning 


an Evaluation 
System 


importance of defining 


professional tasks 


If we stress the importance of the prior 
definition of professional tasks, it is because 
this is a precondition for ensuring that training 
programmes are really designed to meet the 
population’s health needs. Over the last 
10 years or so teachers, under the cloak of an 
educational revival, have used the title of 
educational objectives to disguise what they 
had been in the habit of teaching in the past. 
Such educational objectives have favoured the 
creation or continuation of training pro- 
grammes which only too often seem hardly 
relevant to the needs of the population. 


Indeed, if educational objectives are based 
on faulty principles, then the ‘‘best’’ system 
of training may well give “‘bad”’ results. There 
is even a danger that a ‘‘bad”’ message will be 
‘better’? communicated, and this is certainly 
not the goal sought. 


We therefore propose to demonstrate that 
the professional tasks of a member of the 
health team and the educational objectives 
providing a basis for construction of his 
training programme must be almost identical. 


Another important point to bear in mind is 
that it can be useless to try to change a pro- 
gramme or teaching methods without also 
changing the system of evaluation (parti- 
cularly examinations). Experience has shown 
that if, on the other hand, the evaluation 
system is modified, this has a much greater 
impact on the nature of learning than has 
modification of the programme 
unaccompanied by any change in _ the 
evaluation system. Evaluation provides a 
sound basis for programme planning. There- 
fore, an evaluation mechanism should be set 
up before proceeding to any reform of the 
programme. This makes it possible to measure 
the level at the outset (prerequisite level) 
and the level at the finish and thus to 
determine whether the change has been 
positive or not. This process can be 
represented by what is called the educational 
spiral. 


If the teaching staff are given an opportunity 
to gain the new knowledge they need and to 
acquire the appropriate modern teaching 
skills, they will feel more secure and instead 


of being confined to limited personal 
experience they will accept the use of more 
formal educational research methods. 


This can be a_ powerful stimulus for 
institutional change, particularly when used 
by faculty members whose experience in the 
educational process has already alerted them 
to the ways in which educational innovation 
can be accomplished with the greatest 
possible enthusiasm and the least possible 
hostility on the part of their colleagues. Such 
innovation, based upon carefully gathered 
information and developed according to 
sound educational principles, could enable 
some medical education institutions to 
explore, in particular, non-traditional means 
of preparing the members of the health team 
for the professional tasks they will have to 
undertake. Without the incrusted educational 
tradition that long adherence to a single 
system creates, the opportunity for innovative 
experimentation is far greater. 


This is a very difficult task which may well 
have daunted the most conscientious. We 
consider that teachers should be offered 
assistance in this field. 


That is the main reason why this Handbook 
has been prepared and used during work- 
shops on educational planning. 
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Organizational diagram $s 


howing the relationship between the subsystem “Education” (inside the 


dotted line) and the subsystem ‘‘Health Service’. 
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Analysis of 
Professional Tasks 


Community Health 
Needs and Resources 


Characteristics 
of Country's 
Development 


Health 
Services 


Analysis of Professional 
Functions and Activities 


Characteristics of 
Student Population 
and Teachers 


Planning of 
Evaluation System 


Programme Preparation, 
Choice of Teaching 
Media and Methods 


Implementation of 
Educational Programme 


Formative and 


Certifying Evaluation 


Evaluation of 
Professional Performance 


selection of training goals’ 


Traditionally, this selection has been made 
by relying on the judgment of experts to 
determine what a neophyte in the profession 
ought to know and ought to be able to do. 
In the past we have relied almost exclusively 
on this method. As a result curricula are 
crammed with an ever-burgeoning amount of 
new and highly specialized knowledge which 
the student perceives as irrelevant to his own 
goals and which, in fact, may be of little value 
to other than the super sub-specialist. Certainly 
expert opinion is an important source of 
information about the knowledge and skills 
which trainees should be able to demonstrate, 
but it is also possible to make this decision 
on the basis of scientific evidence about what 
competent health personnel need to know 
and need to be able to do in order to fulfil 
their responsibilities. A number of procedures 
have now been developed for collecting such 
data which provide an empirical basis for 
working out a behavioural description of the 
essential components of professional com- 
petence. This is of great assistance to faculties 
in setting goals and designing curricula. Three 
of these procedures are of special interest: 
the critical incident technique, the method of 
task analysis and the method of analysis of 
epidemiological data. 


The critical incident technique 


This method consists in collecting data about 
specific types of behaviour that characterize 
professional effectiveness and ineffectiveness 
and using these data to make an objective, 
empirical assessment of the essential per- 
formance requirements of the profession. 
This technique is an outgrowth of studies 
in aviation psychology made in the United 
States during World War II. In that 
programme it was found that in reporting 
the reasons for eliminating a trainee, pilot 
instructors and check pilots frequently 
offered such cliches and stereotypes as “‘lack 
of inherent flying ability”, “poor judgment” 
or “unsuitable temperament”. In an effort 
to determine the specific characteristics of 
personnel that contributed to success or 
failure, combat veterans were asked to report 
incidents observed by them that involved 
behaviour which was especially helpful or 
especially inadequate in accomplishing the 


assigned mission. This request concluded with 
the statement: “Describe the officer’s action. 
What did he do?” The several thousand 
incidents submitted in response to this 
inquiry were analysed and categorized to 
provide a relatively objective and concrete 
description of the “critical requirements” 
of combat leadership. 


To apply this method to the health 
professions, several thousand incidents des- 
cribing observations of especially effective 
or ineffective colleague behaviour are collected 
from several hundred health workers 
representing various age groups, geographical 
areas, professional categories and specialty 
interests. For example, in a critical incident 
study of intern and resident performance 
(i.e. of the general practitioner) commissioned 
by the U.S. National Board of Medical 
Examiners, the American Institute of Research 
which conducted the study collected over 
3000 incidents from physicians across the 
country. The incidents submitted involved 
all areas of behaviour: practical, communic- 
ation and intellectual skills. They identified, 
for example, such general requisites of com- 
petence as “Skill in gathering clinical 
information,” i.e., in taking a competent 
history and in performing an adequate 
physical examination, or “Skill in relating to 
the patient and in gaining his cooperation 
in a treatment plan’’. In a similar study con- 
ducted by the University of Illinois Center for 
Educational Development of the critical 
performance requirements in orthopaedic 
surgery, over 1700 incidents were collected 
from more than 1000 orthopaedic surgeons 
representing various practice settings and 
sub-specialty interests. An empirical classific- 
ation defining 94 critical performance require- 
ments, grouped into nine major categories of 
competence, was derived from the incidents. 
This operational and prospective definition 
of the essential components of competence 
could then be used to determine the goals of 
specialty training, the design of programmes 
for their achievement and the criteria and 
methodology for their evaluation. If 


' From “An overview of applied research in medical education 
problems, principles and priorities’. Christine H. McGuire, 
WHO Report on the Workshop on the Needs for Research 
in Medical Education, Alexandria, March 1974. 
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educational planning were regularly based 
on such operationally defined, empirically 
derived goals, educational programmes would 
look quite different. 


Task analysis 


A second method of determining the essential 
components of professional competence which 
should define educational objectives consists 
in detailed task analysis of what various 
categories of health personnel actually do, 
and in deriving from that list of tasks a 
statement of the knowledge and _ skills 
(what should be done, not merely what 
is done) which they must have to perform 
competently. Such a task analysis should 
be based on careful, systematic observations 
of the activities of a representative sample 
of various categories of staff or on the daily 
logs of a representative sample who report 
in minute detail the way in which they spend 
their working days over a specified period 
of time, or on some combination of these 
two approaches. 


Wherever this method has been employed, 
the results have been most enlightening. For 
example, in a limited pilot study of 
paediatricians in a typical small US. city, 
researchers found that all the physicians had 
different but consistent patterns for taking 
a history and performing a physical examin- 
ation. Of the 481 patient visits observed, 
222 were well children; an average of 10.2 
minutes was spent with these children (range: 
7.5 minutes to 13.6 minutes) in contrast 
with an average of 8.1 minutes spent with 
ill children (range: 7.4 minutes to 10 minutes). 
Of the 259 ill children, 104 (i.e. 40%) were 
diagnosed as having an infection of the upper 
respiratory tract, 15 had chronic illnesses 
and five had potentially dangerous diseases. 
For the total group of 481, optic fundi were 
examined only nine times and rectals were 
performed in only six cases; two physicians 
did not percuss the lung fields for any patient. 
The greatest amount of time was spent in 
discussion of nutrition and child development. 
The single most frequent topic on which 
advice was rendered in well-child care con- 
cerned toilet training. The authors of this 


Bergman, A., Probstfield, J. and Wedgewood, R. Per- 
formance analysis in pediatric practice: preliminary report. 
Journal of Medical Education, Vol. 42: 262 (1967). 
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study concluded, “Few aspects of well-child 
care appear to require the skill of a physician 

_ the question is also raised as to whether 
current training programmes are aggravating 
the physician manpower shortage by over- 
training in relation to community health 
needs.’”' 


This is a question that could apply to all 
members of health teams in every country; 
only task analysis or comparable empirical 
studies will give us the answer. 


Epidemiological studies 


One of the most interesting of the newer 
approaches to the use of such studies consists 
in combining three arbitrarily weighted 
factors — disease incidence, individual dis- 
ability and social disruption — to define 
priorities in health care needs and, hence, 
in educational effort. As initially developed 
by Dr. John W. Williamson’ the three factors 
are computed as follows: disease incidence 
consists of a simple tabulation of the 
frequency of the disease (e.g. pneumonia) 
or other medical condition (e.g. pregnancy) 
in the target population. Individual dis- 
ability involves a determination of the extent 
of patient disability or risk associated with a 
given medical condition; an Individual Dis- 
ability Weight (IDW) is calculated for each 
condition from three elements: the average 
length of hospital stay, mortality rates and 
complication rates. Social disruption 
represents an estimate of the disruption 
that would be produced by a given disease 
or condition in the social group of which 
the patient is a member; it is based on such 
factors as cost of illness, age of patient and 
number of dependents, socioeconomic 
standing and the like. For each discharged 
patient a Total Priority Weight (TPW) is 
calculated combining these elements. This 
Total Priority Weight is then arbitrarily 
apportioned among _ patient diagnoses. 
Finally, a cumulative total for each diagnosis 
is calculated from the total patient sample. 
The resultant ranking represents a quantit- 
ative estimate of health care needs or priorities 
for the population at risk. 


2 shes 
Williamson, J. et al. Journal of American Medical Associ- 
ation, Vol. 201: 938 (1967) and Vol. 204: 303 (1968). 


It is clear that even with unlimited resources 
not all of these needs could be met in the 
present state of our knowledge. The next 
step therefore consists of determining what 
portion of total health care needs can be met, 
given our present understanding of disease 
and our present treatment possibilities. 
This portion indicates the target area for 
application of professional skills and helps 
to define educational priorities. The goals of 
education for health service staff can there- 
fore be defined as encompassing those areas 
of health care needs which cause the greatest 
total preventable disability — i.e. those which 
cause the greatest total disruption that could 
be reduced or minimized by early diagnosis 
and appropriate intervention. 


In his early studies using this method to review 
hospital practice in two large community 
hospitals in widely separated metropolitan 
areas in the United States, Dr. Williamson 
found that pregnancy, including uncom- 
plicated delivery, ranked first or second in 
priority in both hospitals, that cerebral 
vascular accidents ranked among the first 
five diagnostic categories in both hospitals 
and that fractures of the lower extremities 
ranked among the first five in one hospital. 
These particular conditions are mentioned 
because in certain educational institutions 
there is a general tendency to reduce the 
amount of clinical instruction for the general 
medical student in some of these areas. For 
example, instruction in orthopaedic surgery 
is often elective despite the fact that trauma 
in general accounts for a very significant 
proportion of total preventable disability. 


While the study reported above was limited 
to hospital practice, the same method could 
easily be applied to any level of health 
practice. Secondly, while the findings from 
such epidemiological studies and the particular 
weights to be assigned to such factors as 
individual disability and social disruption 
will, of course, vary markedly in different 
parts of the world, the approach is clearly 
applicable to any society for which health 
personnel are being trained. 


In all parts of the world, use of such data 
will modify the goals and priorities of 
educational institutions and the emphases 
in curricula by focusing far greater attention 


on ambulatory medicine and on the more 
common causes of disability. 


Implications of applied research 
on goals and priorities 


It can be seen from the above that the means 
are now at hand for supplementing expert 
judgment with data derived from empirical 
studies to assist us in defining the roles and, 
hence, the skills required of students on 
completion of programmes. If such studies 
were carried out as a matter of course and if 
the findings were used to develop explicit 
educational objectives for the health pro- 
fessions, we should see revolutionary changes 
in the kinds of health professionals produced 
and in their training programmes. Further- 
more, such changes would have a far greater 
impact on meeting health care needs than 
would simple expansion of educational 
facilities of the conventional type. 


Here we should mention some simpler but 
also more rapid and less costly techniques 
which can be used to complement or replace 
other methods. These methods are not 
mutually exclusive: 


— Interviews with members of the profession, 
who are asked to describe what, in the light 
of their experience, should be the functions 
and tasks of any member of the health 
team. 


— Questionnaires, made up of either open- 
answer questions (what are the functions of 
... 2?) or closed-answer questions (which of the 
tasks listed below ... ?). 


— The simplest method consists of asking 
each of a group of colleagues to put himself 
in the shoes of a person needing care and 
to describe the functions and tasks that he 
would wish a given member of the health 
services to be able to perform. Comparison 
of the lists submitted will lead to rapid agree- 
ment on a common list of sufficiently high 
quality to provide a basis for a productive 
discussion on the relevance of the programme, 
for example. 


“Transformation of the present professionally 
oriented technologically dominated health system 
into a patient oriented system is the needed 


ingredient for any successful curriculum change. 
The patient should be the primary concern of both 
education and service”’. 


George A. Silver 


example of services 
provided by rural health units* 


population of 5000 persons, normally in one village and 


Each health unit is meant to serve a : 
health team of each of these rural health units 


maybe a few smaller settlements around it. The 
is made up basically of: 
One physician (in charge) 
One assistant midwife 


One assistant sanitarian, and 
One laboratory assistant. 


The rural health unit provides the basic health services for the population it serves, i.e.: 


Maternal and Child Health work 
Communicable Disease Control work 
Vital and Health Statistics work 
Environmental Sanitation work, and 
Medical Care work. 


* Adapted from ‘“‘Three approaches to the analysis of health manpower functions”. HMD/79.1, pp-69 — 72. 


This list was obtained using the questionnaire method and refers to a survey carried out in Egypt in 1969. 


Maternal and Child Health work 


(a) Prenatal Care activities: 


ts 


Conducting abnormal labour, and 
transfer to hospital where necessary. 


1 Comprehensive examination of new 16 cuits BY Doe EE cad a ne 
patients. and intravenous injections. 
2 Follow up examination of patients. (c) Postnatal Care activities: 
Urine analysis (sugar and albumin, 17 Home visiting for puerperal cases. 
microscopic examination). 
18 Detection and treatment of fever 
4 Taking blood samples and determin- wa eX 
ation of haemoglobin percentage. ias it 3 
5 aS 19 Giving hypodermic, intramuscular 
Weighing of pregnant women. as Rai. 
aes Fae and intravenous injections. 
easurement of blood pressure. 
ti ee 20 Carrying out health education and 
7 Prescription of treatment. family planning activities. 
8 Referring patients to hospitals. 
9 Giving hypodermic, intramuscular tacag as Gece eee) 
and intravenous injections. 21 Weighing of children. 

10 Supervision of cleanliness of pregnant 22 Supervision of child cleanliness. 
women. 23 Vaccination against poliomyelitis, 

11 Carrying out health education diphtheria and tuberculosis. 
activities. 24 Temperature taking. 

12 ect : ety 
Home visiting for non-attendants and 25 Carrying out medical examination. 
during the ninth month. ae 

26 Prescribing treatment. 
(b) Natal Care activities: 27 Referring patients to hospitals. 
13 Preparation of delivery bags. 28 Isolation of communicable disease 
14 Conducting normal deliveries at are 


home. 


29 


30 
31 
32 
33 


Giving hypodermic, intramuscular 
and intravenous injections. 


Taking blood samples. 
Circumcision of male children. 
Prescribing the diet. 


Home visiting for non-attendants. 


(e) Miscellaneous technical activities: 


34 
a2 


36 


37 


Preparing the clinic. 


Sterilization of instruments and 


supplies. 


Training of midwives and assistant 
midwives. 


Preparation of drugs for distribution. 


Communicable disease control work: 


(a) Activities related to cases: 


38 
39 


40 
41 


42 
43 


44 


4 (b) 


45 
46 


Isolation of cases. 


Disinfection of cases (during and 
after treatment). 


Dusting of cases (for disinfestation). 


Giving instructions at home 
(education). 
Supervision of domiciliary treatment 


of tuberculosis patients. 


Recording in communicable disease 
register. 


Search for the source of infection. 


Activities related to contacts: 


Surveillance of contacts 


Immunization of contacts. 


(c) General preventive activities: 


47 


48 
49 
50 
51 


52 
53 


Vaccination against poliomyelitis, 
diphtheria and tuberculosis. 

Noting names of non-attendants. 
Preparation of list of families. 
Carrying out periodic dusting. 
Recording in disinfection and dusting 
registers. 

Controlling insects and rodents. 
Carrying out epidemiologic surveys 
for case finding. 


54 


Isolation of detected cases. 


(d) Activities related to deaths: 


Be 


56 


“We 


58 


Receiving notifications of deaths and 
search for relations. 


Examination of the dead and 
establishment of death certificates. 


Recording in the 
registers. 


appropriate 


Issuing of burial permits. 


Vital and health statistics work: 


a 


60 
61 
62 


Recording of births and deaths 
in the appropriate registers. 


Making weekly and monthly reports. 
Calculation of death ratios, etc. 


Making statistical studies and inter- 
pretations. 


Environmental sanitation work: 


63 


64 
65 


66 
67 
68 
69 


70 


71 
(eg 


acs 


74 
75 
76 
77 


Numbering of houses and population 
census. 


Mapping areas and facilities. 


Ensuring cleanliness in and around 
dwellings. 


Hygienic disposal of refuse. 
Constructing latrines in village houses. 
Control of bilharzial snails. 


Identification of breeding places of 
mosquitos. 


Mapping breeding places_— off 
mosquitos. 
Checking hygiene of public latrines. 


Carrying out measures ordered by 
doctor. 


Supervision of — environmental 


sanitation activities. 

Examination of food in public places. 
Taking samples from food. 
Destroying spoiled food. 


Surveillance of market and _ street 
vendors. 


78 


he 


80 


81 


Taking water samples from public 
standpipes. 


laws concerning 


Enforcement of 
cemeteries. 


Examination and certification of 
food handlers. 


out education 


Carrying health 


activities. 


Medical care work 
(a) Diagnosis activities: 


82 
83 
84 
85 
86 
87 
88 
89 
90 
mA 
92 


93 


94 


a 
96 


Preparing the patient. 

Taking the history. 

Recording clinical observations. 
Weighing the patient. 

Taking the temperature. 
Counting the respiration. 
Counting the pulse. 
Measurement of blood pressure. 
Clinical examination. 
Requesting laboratory tests. 


Taking blood samples and administer- 
ing transfusions. 


Microscopic examination of blood 
and blood grouping. 


Urine examination for parasites, 
chemical analysis and microscopic 
examination of urine. 


Requesting X-ray examination. 


Examination of stools. 


(b) Therapeutic activities: 


at 
98 


7? 
100 


Prescribing treatment and/or diet. 


Giving hypodermic, intramuscular 
and intravenous injections and drips. 


Giving oral medication. 


Applying artificial respiration. 


101 
102 


103 


104 
105 
106 
107 


108 
109 


110 
ee 


12 
its 


Catheterization. 


of hot cold 


Application and/or 
compresses. 


Administration of enemas and use of 
stomach pump. 


Suction of mucus. 

Making dressings. 

Eye painting and irrigation. 

Making surgical stitches and per- 
forming minor operations. 

Removal of surgical stitches. 


Carrying out health education and 
supervising patient’s diet. 


Observing patient’s condition. 


Application of external treatment 
(ointment). 


Radiotherapy. 
Physiotherapy. 


Administrative work: 


114 
id 
116 
ibe Mea 
118 


da 
120 
rah 
122 
123 


124 


125 
126 


Assignment of jobs and activities. 
Checking attendance. 

Giving leave permits. 

Conducting legal investigations. 


Management of and 


supplies. 


equipment 


Management of financial matters. 
Filling in forms. 

Book-keeping. 

Correspondence. 


Preparation of monthly and annual 
reports. 


Recording attendance 
room. 


in waiting- 


Supervising housekeeping of the unit. 


Supervising transportation. 


a Pe) a LT a 0 el i ee i fai 


Please Note! 


You are reminded that this list, drawn up in 1969, 
describes the services as they were and not as they 
should have been. It might seem that preventive 
activities deserved greater prominence. 


UU OoODe Roa ooo ob eb oe ae aa 2 


1. Take one category of health personnel (e.g. physician, or nurse, or midwife, or medical 
assistant, or sanitarian) and circle the items on the preceding list corresponding to the 
activities which that category of staff is supposed to carry out in your country at present. 


2. Then think of some activities which that same category does not undertake at present 
but which you feel, in the light of your personal experience, /t should undertake to improve 


the level of health of the population it serves. Draw a square around each of the corresponding 
items on the list. 


3. Describe below any unlisted activities corresponding to the first two questions. 


educational objective 
(derived from professional tasks) 


What the students should be able to do at the end 
of a learning period that they could not do 
beforehand. 


Educational objectives are also called “‘learning 
objectives’ as opposed to “‘teaching objectives’. 


They define what the student, not the teacher, 
should be able to do. 


The definition of the objective of a course is that 
of the result sought, not a description or summary 
of the programme. 


relationship between professional 
acts in the health field and 


educational objectives 


Professional Functions 
or General Objectives 


Professional Activities or 
“‘Intermediate’’ Educational Objectives 


Tasks and Specific 
Educational Objectives 


eee e® 
Rie 
eectetets © 
Coe eets a 
statetets — 
eee e. 


Precise 


Note: oe ses of the circles relates to the number of objectives: the more specific they are 
ore numerous they are. The triangle indicates that at the general level objectives’ 


are ‘‘wide”’ 
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, broad, vague, and that specific objectives are ‘‘punctual’’, narrow, precise. 


types of educational objectives 


1 General objectives: Correspond to the functions of the type(s) of health personnel trained 
in an establishment. 


Example: Providing preventive and curative care to the individual and the community, 
in health and in sickness. 


2 “Intermediate’’ objectives: Arrived at by breaking down professional functions into 
components (activities) which together indicate the nature of those functions. 


Example: Planning and carrying out a blood sampling session for a group of adults in the 
community. 


3 Specific (or instructional) objectives: Corresponding to (or derived from) precise professional 
tasks whose results are observable and measurable against given criteria. 


Example: Using the syringe, to take a blood sample (5 ml.) from the cubital vein of an 
adult (criteria: absence of haematoma; amount of blood taken within 10% of the amount 
required; not more than two attempts). 


To gain better understanding of these three levels of educational objectives and the relationship 
between them, study pp. 1.23 — 1.25 and 1.29 — 1.36. 


data necessary for formulation 
of educational objectives 


A Health needs, demands and resources of society. 
& Services to the patient (list of tasks). 

Bs Service to the community (list of tasks) . 

a The profession itself. 


a The students. 


a Progress in sciences. 
a The scientific method. 
BiG rncns 


For more details refer to: Criteria for the evaluation of learning objectives in the education 
of health personnel. Report of a WHO Study Group. World Health Organization Technical 
Report Series, 1977, No. 608. 47 pages* 

and pp. 4.10 — 4.13 of this Handbook. 


* 


An annex ifi 

ae ai soe hem clarifies what different authors mean by educational objectives, examines the different 

ee. Bec jectives, lists the potential benefits of taking the trouble to formulate objectives and reviews 
ered necessary for this. There is also a short section on how to word objectives properly 
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Take the time to list the main functions of the category of health personnel that interests you 
(dentist, nurse, sanitary engineer, physician, pharrnacist, midwife, etc.). Where possible, refer 
to documents published on the subject in your country (national health plan, professional 
publications, etc.). |f no such data are available, rely on your own experience. 


Se TILEY PESO TLVICELOIIS Oise ve wt eae aa he ea Ar to ke eal eter * are as follows. 


*insert the name of the profession in which you are interested. E.g., “the nurse’’, “the general practitioner’, ‘‘the dentist’, etc. 
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Read the three following 


The professional functions OP ed ea VE A ates ec 
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pages and revise if necessary your own list of functions. 


are as follows: 


pees 
6s BRS i B09 (St et ec GS Sm 8 PN OY A fy! OS CB 


E 
re as who uses a word knows what he means by 
. The problem is that everyone doesn’t realize that 


other peopl i 
a people may have different meanings for the same 


Mager 
SS 50h ES | 
W086 0) OO ta ia said od al 


Primary health care is essential health care based on practical, 
scientifically sound and socially acceptable methods and technology 
made universally accessible to individuals and families in the com- 
munity through their full participation and at a cost that the community 
and country can afford to maintain at every stage of their development 
in the spirit of self-reliance and self-determination. It forms an integral 
part both of the country’s health system, of which it is the central 
function and main focus, and of the overall social and economic de- 
velopment of the community. It is the first level of contact of indi- 
viduals, the family and community with the national health system, 
bringing health care as close as possible to where people live and work, 
and constitutes the first element of a continuing health care process. 


A health team is a group of persons who share acommon health goal 
and common objectives, determined by community needs, towards the 
achievement of which each member of the team contributes, in a coordi- 
nated manner, in accordance with his/her competence and skills, and 
respecting the functions of others. The manner and degree of such 
cooperation will, of course, vary and has to be solved by each society 
according to its own needs and resources. There can be no universally 
acceptable composition of the health team. 
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1.22b 


Primary health care: 


die 


reflects and evolves from the economic conditions and socio- 
cultural and political characteristics of the country and its com- 
munities and is based on the application of the relevant results of 
social, biomedical and health services research and public health 
experience; 


addresses the main health problems in the community, providing 
promotive, preventive, curative and rehabilitative services accord- 
ingly; 

includes at least: education concerning prevailing health problems 
and the methods of preventing and controlling them; promotion 
of food supply and proper nutrition; an adequate supply of safe 
water and basic sanitation; maternal and child health care, includ- 
ing family planning; immunization against the major infectious 
diseases; prevention and control of locally endemic diseases; ap- 
propriate treatment of common diseases and injuries; and pro- 
vision of essential drugs; 


involves, in addition to the health sector, all related sectors and 
aspects of national and community development, in particular 
agriculture, animal husbandry, food, industry, education, housing, 
public works, communications and other sectors; and demands the 
coordinated efforts of all those sectors; 


requires and promotes maximum community and individual self- 
reliance and participation in the planning, organization, operation 
and control of primary health care, making fullest use of local, 
national and other available resources; and to this end develops 
through appropriate education the ability of communities to par- 
ticipate; 


should be sustained by integrated, functional and mutually-sup- 
portive referral systems, leading to the progressive improvement 
of comprehensive health care for all, and giving priority to those 
most in need; 


relies, at local and referral levels, on health workers, including 
physicians, nurses, midwives, auxiliaries and community workers 
as applicable, as well as traditional practitioners as needed, suit- 
ably trained socially and technically to work as a health team and 
to respond to the expressed health needs of the community. 


general objectives 


professional functions 


There will be as many lists of these as there 
are categories of staff trained in the institution 
concerned. 


The following examples of general educational 
objectives at the mstitutional level are real 
sets of objectives as formulated by health 
personnel training institutions. 


They are only examples. Compare them with 
the functions you listed on p.1.21. You may 
find that some of the items are almost 
identical. At this general level the acts 
required to meet the health needs of the 
population will have some points in common 
all over the world. This is quite understandable. 
At this level of general functions it is not 
surprising that nurses, physicians, midwives 
or dentists, for example, should exercise 
similar types of functions, such as treatment, 
prevention, planning, education of the public, 
training of colleagues, etc. The differences 
between the professions will emerge from the 
more detailed list of intermediate objectives, 
describing the activities of each category and 
from the even more specific list of tasks. The 
different types of objectives form a whole. 
They are given meaning by their relationships 
and interdependence. 


What should be noted at this stage is that all 
the examples are relatively short (one page) 
and rather vague. You will also note that 


they define everything the students should 
be able to do at the end of their training. 


They do not define what the teachers do but 
rather what the institution’s “end-product” 
is. They are also known as. institutional 
objectives. 


The fact that the examples that follow are 
numbered does not imply that they are 
classified in order of importance. Obviously 
each function can be more or less important 
as compared with another, depending on the 
health system in which the qualified student 
will work and on the overall stage of develop- 
ment of the country. 


The prominence of the function “health 
education of the public’’ will depend on the 
population’s general level of education. 
Similarly, functions relating to planning 
will be very different depending on the 
development context and degree of organis- 
ation of the country. Physicians and nurses 
in less developed countries may have to 
assume greater responsibilities in this field 
than their counterparts working in more 
developed countries. 


Whatever the relative importance of a given 
function, what counts at this stage is that 
it exists: you will find it useful to bear this 
in mind throughout the training process. 
Now read the examples that follow. 
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Institutional Objectives" ) 


The graduates of the M.D. programme should 
be able: 


1. To identify health problems in their 
totality and to show skills in collecting, 
processing and presenting data pertaining 
to health problems, and subsequently 
to resolve and manage them, from the 
individual level through the family level 
to the community level. 


2. To diagnose and manage frequently 
occurring diseases in the community 
(including emergencies), to identify and 
provide primary care in serious diseases, 
taking account of their physical, 
emotional and social aspects. 


3. To manage health centres at various 
levels and in a variety of settings and to 
work effectively and efficiently in health 
teams, in teaching, research and service, 
with available facilities. 


Institutional Objectives"? 


At the end of his M.D. programme“? the 
graduate will have acquired or developed the 
knowledge, abilities, and attitudes necessary 
to qualify for further education in any medical 
or related health career. The achievement of 
the general goals should enable a student: 


1. To identify and define health problems at 
both an individual and a community 
level and to search for information to 
resolve or manage these problems. 


2. To examine the underlying physical, bio- 
logical and behavioural mechanisms of 
health problems. This includes a spectrum 
of phenomena from the molecular to 


those involving the patient’s family and 
community. 


3. To investigate community health problems 
and to recommend efficient and effective 
approaches to deal with environmental, 
occupational, behavioural, and public 
policy issues. 


4. To develop the clinical skills and methods 
required to define and manage the health 
problems of patients, including their 
physical, emotional, and social aspects, 
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(1) 


To apply basic principles in health 
education in order to assist and lead the 
planning, implementation and evaluation 
of health programmes in promoting 
health, preventing disease, cure and 
rehabilitation, according to the needs of 
the community and local social, religious, 
customary and cultural values which can 
influence the state of health and disease. 


To identify personal limitations, and to 
nurture the capacity and interest in 
enhancing their knowledge and develop- 
ing personal characteristics required for 
professional advancement through an 
awareness of personal assets and limit- 
ations. 


To function as an effective and efficient 
member of a team with a sense of 
responsibility and dependability. 

Adapted from Gadjah Mada University Faculty of 
Medicine, Yogyakarta, Indonesia. 


within the context of effective health 
care. 


To recognize, maintain, and develop the 
personal characteristics and attitudes 
required for a career ina health profession. 
These include: 


a. Awareness of personal assets, limit- 
ations, and emotional reactions. 


b. Responsibility and dependability. 


c. Ability to relate to, and show concern 
for, other individuals. 


To be a self-directed learner, recognizing 
personal educational needs, selecting 
appropriate learning resources, and 
evaluating personal progress. 


To assess critically professional activity 
related to patient care, health care delivery, 
and health research. 


To function as a productive member of a 


small group which is engaged in learning, 
research, or health care. 


To work in a variety of health settings. 


1 : ¢ 
(1) from McMaster University, Canada. 


Institutional Objectives‘! ) 


Description of Practitioner of Tomorrow 
(Nursing) 


The graduate of the new baccalaureate nursing 
programme will be prepared to function as 
a generalist with beginning competencies in 
a specialized area of nursing. 


The graduate will be prepared to function in 
a variety of settings and be able to: 


1. Obtain health histories and make general 
health assessments. 


2. Provide safe and competent care in 
emergency situations and acute illnesses. 


3. Provide supportive care to persons with 
chronic or terminal health problems. 


4. Provide health teaching, guidance and 
counselling. 


5. Assist persons 
health status. 


to maintain optimal 


Provide for continuity of health services. 


Assume leadership responsibility for 
planning and evaluating nursing care. 


8. Work effectively with all persons con- 
cerned with health care problems. 


This baccalaureate nurse, as a practitioner of 
nursing, will be accountable and responsible 
to clients for the quality of nursing whether 
administered directly or indirectly. 


From the University of Washington School of Nursing. 


i 


These three examples of general educational 
objectives are from both developed and 
developing countries. 


What is striking is the overall similarity 
between the functions listed, even if they 
are expressed in different terms: 


= providing treatment 
providing preventive care 


E 
= planning services (management) 

w health education of the population 
@ 


collaboration with other services in the 
interests of overall development 


training of health personnel 
participation in research 


evaluation of own activities 


continuous development of own skills 


They were gathered during a recent world- 
wide survey designed to collect general 
educational objectives for nurses and 
physicians. Identical results were obtained; 
that is, the same functions came up in 
practically every case. This collection of 


functions corresponds to the role that health 


services personnel are expected to fulfil. 


Throughout this Handbook you will be 
invited to use this list of functions (and others 
if necessary, depending on the health needs 
of the population in your country) as the 
basis for your future educational decision- 
making. 


To demonstrate how such an obvious list 
can in fact be a very effective instrument, 
you are invited to use it at once in making 
a quick analysis on the next page. 
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Personal Notes 


1.26 


Tricky test to force you to think about the relevance of a programme 


Take the functions you listed on p.1.21 


For each function that corresponds to one of those listed in the table below, ask yourself the 
following two questions: 


In the institution where | work 


ie 


are teaching activities organized to help students acquire skills corresponding to each function 
listed? : 


do the examinations (counting towards award of diploma) effectively measure the students’ 
abilities in relation to each function listed? 


Where you can answer YES (without blushing), mark a cross in the + column. 
If no corresponding activity is organized by your institution, mark a cross in the O column. 
If you are not sure, mark a cross in the ‘‘+ or -’’ column. 


Teaching activities that Examinations providing 
help student to perform effective measurement 
function of function 


Health Education 
Collaboration 


Training of 


other categories 


Self-training 


General educational objectives 
provide a useful basis for preparation 
of a relevant programme 
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professional activities 


and intermediate objectives 


Intermediate educational objectives are 
obtained by breaking down each function (or 
general objective) into smaller components. 
These components are professional activities 
which in their turn can be broken down 
into more specific acts that are called 
professional tasks as long as they can be 
measured against given criteria. (See p. 1.35 
et seq. for specific objectives). It can also be 
said that all objectives that are neither general 
nor specific are on the intermediate level. 
That is, there can be several intermediate 
levels rather than a single one. 


The pages that follow give examples of 
intermediate objectives'. Unlike general 
objectives, whose vagueness makes them fairly 
universal, intermediate objectives reflect the 
health needs of a population living in a given 
context. This restrictive list of diseases, 
therefore, would have been different in the 
case of a general practitioner in Finland or 
in Cameroon because of the special geo- 
graphical epidemiology of each country. 
The social and political system and the type 
of health services provided will also have an 
influence. These are the factors that ensure 
the relevance of educational objectives. 
Another important point to be taken into 
consideration: this list, like any other list 
of educational objectives, is only a means 
or working instrument and not an end in 
itself. It was drawn up as a basis for choosing 
instruments of evaluation for measuring the 
skills of students during their internship. 

; Prepared by a multidisciplinary group of teachers from 


Algeria. Workshop on docimology, Timimoun, Algeria, 
February 1977. 


Examples 
of intermediate educational objectives 


These intermediate educational objectives 
were derived from general objectives defining 
the functions of a general practitioner. 
They refer to the paediatric aspect of his 
work. 


The general practitioner should be able to 
carry out the following activities: 


1. Diagnose and treat major childhood 
disorders: 


abnormal development of the embryo 
or foetus 

infections in newborn babies 
emergency surgery on newborn babies 
jaundice of the newborn 
vomiting in infants 

cardiac insufficiency 

acute diarrhoea 

dehydration 

convulsions 

purulent meningitis 

tuberculous meningitis 
tuberculosis 

eruptive fevers 

viral bronchopneumonia 
bacterial pneumonia 

septicaemia 

childhood skin disorders 

urinary infections 

acute glomerular affections 
abdominal tumours 

enlargement of liver 

enlargement of adenoids 
enlargement of spleen 

kala-azar 

malaria 

throat infections 

Otitis 

orthopaedic problems in children 


Carry out activities relating to patient 
care, taking of samples, laboratory work 
and use of equipment. 


2.1 Sampling techniques: 


blood (including blood from umbilical 
cord) 

abcess 

cerebrospinal fluid (CSF) 

urine 

puncture of ascites, pleura 


2.2 Techniques relating to patient care, 


preventive measures and laboratory 
work: 

immunization 

perfusion, intramuscular 
transfusion 
catheterization, enema 
blood grouping 
haematocrit 

erythrocyte sedimentation rate 


injection, 


1.29 


— CSF count 


2.3 Use of equipment: 


— sphygmomanometer 
— otoscope 

— aerosol spray 

— aspirator 

— electrocardiograph 
— ophthalmoscope 


. Distinguish between normal newborn 
babies and those at risk; organize 
prevention and early detection of possible 
dangers. 


3.1 Recognize growth anomalies. 


3.2 Recognize anomalies of psychomotor 
development. 


3.3 Work out with the parents a diet 
suitable for the needs of their child. 


3.4 Recognize dietary anomalies. 


3.5 Plan a surveillance programme for a 
normal child and for one at risk. 


3.6 Enter findings in the child’s medical 
record. 


. Plan, in collaboration with the parents, 
individual and collective surveillance of 
growth, nutrition and psychomotor 
development in children (newborn babies, 
infants, children). 


4.1 Plan care of a normal newborn baby. 


4.2 Plan treatment of a newborn baby 
with a diabetic mother. 


4.3 Plan treatment of a Rh negative new- 
born baby. 


4.4 Plan treatment of a newborn baby 
with kidney disease. 


4.5 Plan treatment of a newborn baby 
with low birth weight. 


4.6 Plan treatment of a premature baby. 


4.7 Plan treatment of a baby born after 
abnormal labour. 


Identify somatic problems (particularly 
relating to growth and nutrition), psycho- 
motor and emotional problems in a sick 
child on the basis of medical history and 
clinical examination. 


5.1 Question parents of a sick child and 
make a record of the information 


obtained. 
5.2 Examine a sick child. 
5.3 Make a note of the findings. 


5.4 On the basis of a clinical examination, 
determine the problems presented 
by a sick child (particularly relating 
to growth, nutrition and psycho- 
motor development). 


Protect children individually and 
collectively against the effects of 
hereditary conditions, communicable 
diseases and accidents. 


6.1 Offer advice on genetic matters to 
parents. 


6.2 Determine the mode of transmission 
of hereditary diseases. 


6.3 Detect and treat hereditary diseases. 


6.4 Investigate home conditions of a child 
with a communicable disease. 


6.5 Examine contacts of a child with a 
communicable disease and apply 
preventive measures. 


6.6 Carry out all immunizations. 


6.7 Draw up a schedule for a child never 
or inadequately immunized. 


6.8 List, in order of frequency, the 
accidents that happen to children in 
a given sector. 


6.9 Organize and participate in a cam- 
paign to prevent accidents to children. 


Identify mental health problems in 
children; propose measures and participate 
in their application. 


7.1 Determine the priority mental health 
problems in children of his own 
health sector. 


7.2 Coordinate health, administrative and 
educational resources available for 
dealing with mental health problems 
in children (particularly those relating 
to maladjusted or abandoned children). 


Evaluate the effects on child health of 
the environment; propose appropriate 
measures and ensure that they are applied, 
individually and collectively. 


10. 


8.1 List environmental factors in_ his 
own area of work. 
8.2 Help improve environmental 


conditions in collaboration with the 
health authorities. 


8.3 Identify a child seriously threatened 
by his environment. 


8.4 Detect and treat a disorder caused 
by the environment. 


8.5 Advise parents on drawing optimum 
benefit from a favourable environ- 
ment. 


Be accessible to the child and his family, 

providing health education and _ the 

support needed in case of disease or 

disability. 

Organize his plan of work to ensure that: 

9.1 He is accessible to the child and his 
family. 

9.2 He has time to listen to them. 

9.3 He has time to talk to them. 

9.4 He has time to reassure them. 

9.5 He has time and the ability to 


provide the child and his family with 
the necessary health education. 


Organize prevention, detection and 
follow-up of deficiency diseases and 
chronic conditions. 


10.1 Apply national regulations for the 
prevention of deficiency diseases. 


LOI Detect 
deficiency diseases in a 
population: 


and treat the following 
given 


— protein and calorie malnutrition 
marasmus 
kwashiorkor 


— hypovitaminoses 
vitamin D deficiencies 
(rickets) 
vitamin A deficiencies 
(hemeralopia, xeroma) 
vitamin B complex deficiencies 
(beriberi, pellagra, megaloblastic 
anaemia) 


vitamin C deficiencies 
(scurvy) 

— iron defictency 
(anaemia due to lack of iron). 


11. 


10.3 Detect and treat chronic con- 


ditions in children: 


_ diabetes 

_ haemoglobinopathy 

os thalassaemia 

_ rheumatic conditions 

_ metabolic disorders 
(phenylketonuria, glycogenosis, 
glucose 6-phosphate dehydro- 
genase deficiency, galactosaemia) 

— congenital or acquired heart 
conditions 

- chronic respiratory insufficiency 
(mucoviscidosis, bronchial dilat- 
ation, deformations of the thorax, 
asthma... ) 

oe epilepsy 

= haemophilia 

=< chronic allergic conditions 
(eczema, allergies in the upper 
respiratory tract, asthma .. .) 


Organize, participate in an_ evaluate 
treatment and _ preventive activities 
(medical and otherwise). 


11.1 Allocate tasks among members of 
a health team in his area of work. 

11.2.1 Carry out a paediatric consultation. 

11.2.2 Decide to admit a patient to 
hospital. 

11.3. Work in a ward as part of a team. 

11.4 Organize reception and surveillance 
of emergency cases. 

11.5 Take part in the activities of a 
maternal and child health centre. 

11.6 Deal with problems relating to 
drugs and equipment. 

11.7 Help organize an immunization 
campaign. 

11.8 Propose and ensure application 
of non-medical measures 
required to back up medical 
activities in the field of pre- 
vention and hygiene. 

11.9 Set up a mechanism of periodic 


evaluation by all team members 
of his own and the team’s activities, 
in terms of their objectives. 


12. Help families to use health and adminis- 
trative bodies concerned with improving 
child health. 

12.1 Ensure health coverage of ll 
children living in his area. 

12.2 Enumerate the health structures that 
exist in his area. 

12.3 Enumerate the administrative, 
political and economic bodies in his 
area. 

12.4 Assess the role played by each of 
the preceding in improving child 
health. 

12.5 Organize optimum use of health 


Lor 


facilities. 


12.6 Promote and enforce measures aimed 
~ at improving child health. 


Plan training and retraining of health 
staff. 


13.1 Remedy any gaps in the skills of 
members of his health team. 


13.2 Organize the training and/or 
continuous education of members 
of the health team. 


13.3 Evaluate the training and further 
education activities of health team 
staff. 


Lis2 
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The definition of educational objectives has 
become almost a fashionable subject of con- 
versation . . . defining them is becoming a mark of 
modernity . . . but, as with all slogans, there is 
a danger that we shall get used to them without 
understanding their purpose, their nature, their 
advantages, their limitations and the risks involved. 


In drawing up educational objectives, what counts 
is not their formal definition but their relevance to 
the professional tasks of the personnel to be 
trained. 
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Before going any further, think about the 
concept of relevance and about the fact 
that educational objectives are a means and 
not an end. 


It seems reasonable to believe that an 
educational programme has more chance of 
being effective if its purposes have been 
clearly expressed. Experimental research in 
the field of evaluation indicates that it is 
not possible to measure the results obtained 
from an educational system if its objectives 
have not been explicitly defined. But a desire 
for precision should not divert us from a 
much more important concern: the need for 
relevance. 


To be relevant an educational programme, 
rather than being the result of a non-selective 
mass of knowledge accumulated over the 
centuries, should be selectively shaped in 
terms of the aims to be achieved. Each 
time the goal is modified, the programme 
too must be modified accordingly. 


Thus relevance is the degree of conformity 
that exists between training programmes 
on the one hand and the population’s health 
needs and resources on the other. 


When determining educational objectives, 
therefore, we must take into account the 
health needs and resources of society, the 
health professions, the progress of science, 
the capabilities of the students, the social 
and cultural context, etc. Moreover, the 
study made of these factors must be 
prospective’ in nature since we are training 
personnel for the future. 


The method traditionally used is to bring 
together eminent professors and the result 
of their deliberations is presented as a list of 
chapter headings. Often, existing 
programmes are used as the main source of 
data for the preparation of the new pro- 
gramme. The professors indicate the number 
of hours to be devoted to the various subjects 
to be dealt with: this generally leads to a 
conflict of personalities and it is the most 
forceful, the most persuasive, sometimes 
the most irascible or noisiest of the 
participants in the discussions who obtains 
the largest number of hours. The result is 
that the time factor becomes a constant and 
that competence remains an undefined 
variable ,* 


In the absence of a definition of relevant 
educational objectives, discussions on pro- 
grammes, teaching methods and evaluation 
methods are difficult and often futile. 


When educational objectives have been 
established at the intermediate level (lists 
of activities), it is then possible to determine 
with some precision which learning activities 
are likely to facilitate the attainment of an 
objective and which are not. 


Methods of evaluation (of students, teachers 
and methods) will also depend on the 
objectives to be achieved. Evaluation 
consists in being able to say to what extent 
and how the objective set has been achieved. 
If one has not bothered to lay down a 
measurable objective, it will be difficult to 
make any kind of evaluation. To select a 
type of examination (to use an old-fashioned 
term) without educational objectives makes 
no more sense that to try to choose a 
measuring instrument without knowing what 
has to be measured. Only over the last two 
decades have investigators in the field of 
testing and measurement begun to work out 
a solution to this problem. Here, too, the 
weight of tradition and emotional reactions 
make themselves felt. 


ipTie epidemiological, sociological data, etc., and the 


operational research necessary for such a prospective analysis 
are not dealt with in this Handbook. For information on 
these matters the reader should consult specialized public- 
ations dealing with the organization of health services (see 
Bibliography, p. 7.01 et seq.). 


2 For example, the length of medical studies is fixed by 


administrative regulations at 6 years (or 7, 5 etc.): this is 
the time constant. On the other hand, there is hardly any 
definition of the competence of graduates, and this can 
lead to great variability. 


Built-in relevance approach to educational planning 


Community Health Problems 
Health Care Programme 
Organized Health Services 


| “Role’’” 


Corresponding terms 
used 
in Health Services 


Corresponding terms 
used 


in Education Services 


Role 
Goals/Aims 
General Learning | | iG, | Functions 
Objectives “Functions of Health Are too 
— _(Institutional) si | Personnel often used 
. ““Activities’’” Activities as synonyms 
Intermediate Learning | 

jectives 

as oa Tasks 
‘“*Tasks’ 


Communication 
Skill 


Specific Learning 
Objectives 


Examinations 


Practical 


Skill 


Certifying Assessment 


Evaluation 


some definitions * 


Role: Pattern of motives and goals, beliefs, 
values, attitudes and behaviour (played by a 
person in society) which one has to assume 
(0); functions performed by someone (w); 
“the role of a teacher’’. 


Function(s): A set of activities, expected of a 
person by virtue of his position. (The function 
of a chairman is to ‘‘preside over meetings’’) 
(w). 


Activity: A combination of specific tasks 
whose fulfilment leads directly to the achieve- 
ment of a function. 


Task(s) Any piece of work that has to be done 
(0); a specific piece of work expected to be 
finished within a certain time; a set of actions 
necessary to the fulfilment of a given activity. 


1.34 


Competence (y) Ability required to carry out 
a task; sufficiency of qualification (o); the 
quality of being functionally adequate or 
having sufficient skill for a particular function; 
competence is a potential which is realized at 
the moment of performance. 


Performance The carrying out of a task: the 
doing of any action or work; the execution of 
an action; 


Action: The process of doing, working (0). 


Skill Practical knowledge in combination 
with ability (0). 


dexterity in the execution of learned physical 
or intellectual tasks (w). 


“Sources: Oxford English Dictionary (o), Webster (w), 


professional tasks and 
specific educational objectives 


Having established the principal functions 
that outline the role of a health worker, we 
then went on to achieve a greater degree of 
precision by describing the activities corres- 
ponding to each function. Now we must go 
further and define each of the specific 
professional tasks corresponding to each 
activity. Let us continue with the example 
furnished by our Algerian colleagues. 


They decided to define the tasks corres- 
ponding to activity 11.4: Organize reception 
and surveillance of emergency cases’ (see p. 
POF 


Here is the list: 


11.4.1 Check the availability of equipment 
needed for emergencies (drugs, 
instruments, beds), using a checklist. 

11.4.2 Treat, in order of urgency, several 
patients who arrive at once. 

11.4.3 Support the vital functions of a 
child, in accordance with an ad hoc 
outline of procedures. 

11.4.4 Avoid any action that could endanger 
the life of the child. 

11.4.5 Handle the child gently. 

11.4.6 Reassure the child. 

11.4.7. Explain to the parents why the 
child must be kept in hospital. 

11.4.8 Offer moral support to the parents. 

11.4.9 Organize a surveillance schedule for 


an emergency Case. 


11.4.10 Decide to move the patient. 


11.4.11 Plan the move. 
11.4.12 Prepare a newborn baby for transfer. 
11.4.13 Prepare a child for transfer. 


11.4.14 Explain to the parents how the 
administrative structures involved in 
admissions and departures function. 


11.4.15 Identify the various administrative 
structures involved in a referral. 


11.4.16 Distribute work among health per- 
sonnel assigned to the emergency 
service. 


11.4.17 Elicit the reasons for various sur- 
veillance activities from nursing staff. 


11.4.18 Explain the reasons for various sur- 
veillance activities to nursing staff. 


11.4.19 Reassure the mother of a child 
admitted to hospital. 


The definition of educational objectives 
must result from a collective effort, in which 
the students have a part to play. 


Any hopes that teachers may have of 
developing the motivation of students will 
be realized only when the latter have really 
been able to make a choice, that is, to decide 
for themselves on the tasks (educational 
objectives) to be performed. For this the 
tasks (objectives) must be clearly defined 
and presented as a choice; it is even better 
if the students can participate in their 
formulation. 


identifying the components of a task 


An analysis of the tasks listed above shows 
that performance of some tasks involves 
not only practical skills (e.g., handle the 
child ...., 11.4.5) but also communication 
skills (same example; handle the child gently) 
and, finally, intellectual skills (e.g., distribute 
work among health personnel, 11.4.16). 
All three components are involved in some 
tasks; in others, only two (e.g., 11.4.5 or 
11.4.16) and sometimes only one (11.4.11). 
In some cases the various components 
(practical, communication and _ intellectual 


skills) overlap to such an extent that it is 
possible only to identify the one that pre- 
dominates. For example, “Offer moral support 
to the parents” (11.4.8) is principally a 
matter of attitude (communication skills). 
Of course, in some cultural contexts this 
could be expressed by an act, a gesture or 
a tone of voice. To know what attitude 
to adopt, previous knowledge of the milieu 
and its customs is obviously necessary. 
Another example is “Decide to move the 
patient” (11.4.10). This is primarily an 
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“intellectual” problem. The decision 


(intellectual activity) will be taken on the 
basis of the information available at a given 
moment. Of course, that information will 
have been obtained by means of practical 
skills (physical examination of patient) and 
the human aspect of the problem cannot be 
overlooked. To sum up, it can be said that 
a professional task will often be represented 
by practical skills. Since those practical 
skills usually involve another person (the 
patient, his family, a colleague), a certain 
attitude (communication _ skill) will be 
appropriate. Lastly, some measure of 
intellectual skill (knowledge) will be needed 
for performance of the task. 


Identification of all the components of each 
task is therefore recommended, to ensure 
that each component is taken into account 
during the successive stages of the educational 
process. The learning process and teaching 
activities will be designed to facilitate assimil- 
ation of each component, and the same will 
apply when the time comes to evaluate the 
students. 


The classification that follows will help you 
by providing more details about the three 
domains of practical, communication and 
intellectual skills. 
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If the jargon seems obscure, remember to consult 
the Glossary (p. 6.01 et.seq.). 
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classification of educational objectives 
into three domains: practical, 
communication and intellectual skills’ 


To help teachers in precise formulation of 
educational objectives, systems of classifi- 
cation into domains (practical skills; attitudes 
or communication skills; intellectual skills 
and knowledge) and within each domain 
(different levels of the process) have been 
put forward by various specialists in education 


and psychology. 


One of their reasons was to try to clear up 
the confusion resulting from the use of 
ambiguous objectives such as ‘‘The student 
should have a good understanding (or know- 
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For some, “good understanding of the law 
of X’’ may mean that the student should be 
capable of giving the mathematical formula 
for the law or saying what it means, and for 
others that he should also be able to use the 
formula to solve a problem or show the 
interdependence of the law in relation to 
other phenomena. No one questions the need 
for students to have a ‘“‘good understanding”’ 
of a particular phenomenon, law or principle, 
fact or theory. However, what satisfies one 
teacher and enables him to say whether 
a student has in fact achieved a ‘“‘good under- 
standing’? may be very different from what 
would satisfy another teacher, unless they 
have agreed on what they mean by “good 
understanding’ through the use of 
unambiguous terminology. 


Education studies have often been criticized 
for the imprecision of their terminology and 
concepts. The chief value of a classification 
is that it enables educators to communicate 
more precisely among themselves. 


This improved understanding will enable 
those responsible for programme construction 
to work together. A classification system 
could also be of use in evaluating the results 
of a system of education. 


Definition and evaluation of the quality of 
the instruction in a school, country or region 
poses a very difficult problem; and a well- 
designed system of classification can play 
an important role in finding a solution. 
Another important function is to help 
educators to prepare examinations for students 
who have received instruction. A classification 


is also useful in general as a research tool in 
education and evaluation. 


We do not feel that strict classification is 
possible. Overlapping of adjacent categories 
will occur everywhere, as is the case in all 
aspects of human thought and activity. What 
counts is not to label but to understand and 
to develop a common language. 


We do not propose to give a detailed account 
of the various classification systems put 
forward by  B.S.Bloom, D. Krathwohl, 
R.Gagne or E. Thorndike. We shall limit 
ourselves to presenting an appropriate and 
simplified classification for each of the three 
domains. 


1. Domain of attitudes (communication 
skills). The dictionary tells us that an attitude 
is ‘“‘behaviour representative of feeling or 
conviction; a persistent disposition to act 
either positively or negatively towards a 
person, group, object, situation or value”’. 
(Webster). 


For the purposes of this Handbook we shall 
restrict this domain to everything concerning 
relations between health personnel and the 
members of the community they will serve. 
It will therefore be almost entirely a matter 
of interpersonal relations. This is why it is 
referred to as a communication skill. Three 
levels can be distinguished. 


1.1 The first is the level of Receptivity or 
attention; it implies sensitivity to the existence 
of a certain phenomenon and includes a 
willingness to receive. 


Example: Noticing the anxiety of a patient 
awaiting the result of a laboratory test for a 
disease that can have serious consequences. 


1.2 The second level is that of Response. 
This implies sufficient interest in the pheno- 
menon noticed to do something about it. 


Example: In the case described in the previous 
example, the response would be to say a few 
reassuring words to the patient so that he 
does not feel alone. 


' Summary of an article by Dr. J.-J. Guilbert, Didakta 
Medica, February 1971. 
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1.3. The third level is that of Internalization. 
This implies that your perception of a pheno- 
menon has found a place in your scale of 
values and has affected you long enough for 
you to adapt yourself to the value system 
of the other person. This enables you to 
adapt your attitude to the other person as if 
you were experiencing the same phenomenon 


yourself. 


Example: On the death of a child, your 
attitude to members of his family will show 
them that you care about their grief and 
are ready to help them to get over it. This 
does not mean that you have internalized 
their grief but that you have internalized the 
attitude that enables you to offer them 
effective help. 


2. Domain of practical skills 


By this we mean the routine actions carried 
out by health workers (intramuscular 
injection, lumbar puncture, etc.). 


Three levels can be distinguished. 


2.1. The first level is that of Imitation. The 
student, exposed to an observable action, 
makes an attempt to copy it step by step, 
guided by an impulse to imitate: he needs 
a model. 


Example: A student nurse who has seen 
intramuscular injections performed before 
her several times tries to imitate the move- 
ments involved . . . using an orange. 


2.2 The second level is that of Control. At 
this stage the student is able to demonstrate 
a skill according to instructions and not 
merely on the basis of observation. He also 
begins to differentiate between one set of 
skills and another and to be able to choose 
the one required. He starts to be adept at 
handling selected instruments. 


Example: The student becomes accustomed 
to carrying out certain acts while performing 
routine minor surgery under supervision. 


2.3 The third level is that of Automatism. 
A high degree of proficiency is attained in 
using the skill, which now requires only 
a minimum of energy. 


Example: An experienced nurse washes a 
bed-ridden patient carefully and without 
causing any discomfort; or a physician deftly 
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‘ntubates a road accident victim in the midst 
of the surrounding confusion. 


3. Domain of intellectual skills 


In 1963, C. McGuire’ proposed a classifi- 
cation system derived from that of Bloom? 
and designed more specifically for use in the 
preparation of achievement tests for students. 


Levels 
1. Knowledge 
1.1 Recall 


1.2 Recognition of meaning 
2. Generalization 


Solving of a routine problem 
3.1 Interpretation of data 
3.2 Application 


4. Solving of an unfamiliar problem 
4.1 Analysis of data 
4.2 Special application 


5. Evaluation 


6. Synthesis 


Three levels are probably enough for the pur- 
poses of defining educational objectives and 
student evaluation. The three levels are taken 
from McGuire’s system. 


1 Recall of facts. 
2 Interpretation of data. 
3 Problem solving. 


3.1 The first level is that of Recall of facts. 
This involves remembering the facts, 
principles, processes, patterns and methods 
necessary for efficient performance of a pro- 
fessional task. 


Example: The student must be able to 
converse with his fellows and his teachers 
using professional language which he should 
understand without constant reference to a 
dictionary or other text; or communicate 
orally with members of society; or com- 
municate in writing (scientific articles, reports, 
findings, etc.). 


3.2 The second level is that of Interpretation 
of data. This is a process of application or 
use of ideas, principles or methods to deal 
' Centre for Education Development, University of Illinois 


Medical School. 


2 

Bloom, B.S. (ed.), Taxonomy of Educational Objectives, 
Handbook I: The Cognitive Domain, David McKay, New 
York, 1956. 


with a new phenomenon or situation. 


Example: After analysing observable data, 
the student interprets their meaning, grasps 
their relationships and arranges them into a 
known pattern. 


3.3 The third level is that of Problem solving 
(relating to diagnosis, treatment, organization, 
etc.). This at best should include finding 
solutions for a problem arising from new 
situations with no precedent to serve as a 
guide. 

Example: The student, faced with a patho- 
logical condition for which he has not been 
prepared, is able to get on the right track 
by applying scientific methods and a sound 
experimental approach. 


The only purpose of these classification 
systems is to permit analysis of the learning 


process and to help teachers in educational 
decision-making. It is obvious that human 
behaviour can rarely be divided neatly into 
practical, communication and _ intellectual 
skills. The same division in objectives is some- 
what artificial. Neither teachers nor those 
responsible for preparing programmes can 
separate them entirely. 


These differences will remain artificial as long 
as we lack the instruments of evaluation 
needed for simultaneous study of learning 
experiences in the various domains (practical, 
communication and intellectual skills). It is 
to be hoped that explicit definition of 
educational objectives will gain in importance 
so that research workers in education can 
solve certain problems and the choice of 
appropriate teaching methods will be made 
easier. 
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The coding of information for scientific purposes 
involves reduction of the individual to a few basic 


characteristics. 


Beware! Oversimplification of data can lead to absurd 


conclusions. 
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the three classifications (simplified) 


able to solve a new 
problem 


able to interpret data 


able to recall facts 


domain of intellectual 
skills 


Note: 
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able to internalize 
a feeling 


able to supply a 
response to the affective 
need of another person 


able to show receptivity 
towards another person 


domain of 
communication skills 


participation in 
educational activities 


able to perform a practical 
act automatically and with 
a high degree of efficiency 


able to exercise effective 
control over the practical 
skill 


able to imitate the 
actions of a model 


domain of 
practical skills 


Please remember that in human behaviour the three domains are often intricately connected. 


Indicate by a cross the components [practical skill (P), communication skill (C) or intellectual 
skill (I)] of each of the 19 tasks listed below. 


Mark two crosses in the box corresponding to the dominant component. 


Domains 


han o a List of Professional Tasks 


oe 1. Check the availability of equipment needed for emergencies (drugs, 


instruments, beds), using a checklist. 


eae 2. Treat, in order of urgency, several patients who arrive at once. 


3. Support the vital functions of a child, in accordance with an ad hoc 
outline of procedures. | 
ae | 4. Avoid any action that could endanger the life of the child. 
feslen 5. Handle the child gently. 


eee 6. Reassure the child. | 
[|_| 7 explain tote parents why the eid must be kept ines 
P| | | offermortupportto themes, 
ees 9. Organize a surveillance schedule for an emergency case. 


10. Decide to move the patient. 


fase 11. Plan the move. 
Balch lee 12. Prepare a newborn baby for transfer. 


Pee 13. Prepare a child for transfer. 


a 14. Explain to the parents how the administrative structures involved in 


admissions and departures function. 
ee 15. Identify the various administrative structures involved in a referral. 
Cee 16. Distribute work among health personnel assigned to the emergency service. 
wi 17. Elicit the reasons for various surveillance activities from nursing staff. 


| pale a 18. Explain the reasons for various surveillance activities to nursing staff. 


nn 19. Reassure the mother of a child admitted to hospital. 


Compare your answers with those of the author of the Handbook, page 1.51. 
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definition of specific 
educational objectives 


— Refer to the example of a task (p.1.19): 


“Using a syringe, take a blood sample (5 ml) 
from the cubital vein of an adult’”’. 


Qualities of a 
Specific Educational 
Objective 


— Break it down into components. The main 
component is a practical skill, but the other 
two (communication and intellectual skills). 
are also involved. 


— The criteria proposed in this example 
(absence of haematoma; amount of blood 
taken within 10% of the amount required; 
not more than two attempts) confirm that the 
person who set the task considered the 
practical skill to be its principal component. 


Relevant 


— This type of act occurs frequently in the 
daily routine of health workers. It resembles 
the administration of intramuscular injections, 
lumbar punctures, puncture of ascites, pleura 
or articulation, gastric intubation. 


Logical 


— The communication skill appropriate to all 
these acts will be the same. It can be defined 
by acceptable types of behaviour which, when 
described in an observation table, can be used Unequivocal 
as criteria (introducing yourself to the patient, 
making sure that he knows what you are 
going to do, etc.). 


— For all these acts a certain amount of Feasible 
theoretical knowledge or itellectual skill 
is needed (anatomy, sterilization procedures, 
secondary effects, etc.): this is usually the 
prerequisite level which in its turn can be 
defined in terms of specific contributory Observable 
educational objectives: (also called enabling 
objectives): 


— sketch from memory the position of the 
cubital vein; 


— list the measures to be taken to ensure 


sterility during the act. Measurable 


(in both cases criteria should be established 
from a textbook). 


To sum up, a specific educational objective 
can be defined as a task accompanied by a 
criterion indicating an acceptable level of 
performance for its principal component. 


Specific Objective = Task + Criterion 
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What are the qualities of a specific educational 
objective? 


It must be all of the following: 


= Relevant: Its definition should be free of 


any superfluous material but cover every 
point relating to the aims in view, 1.e., the 
general objectives derived from the health 


needs of society. 


Unequivocal: “Loaded” words (words 
open to a wide range of interpretations) 
should not be used, to avoid any possibility 
of misunderstanding. What do we mean 
when we say we want a student to ‘“‘know”’ 
something? Do we want him to be able to 
recite, or to solve, or to construct? To 
say merely that we want him to “know”’ 
tells him too little or too much. The 
objective is unequivocal when you describe 
what the learner will have to do to 
demonstrate that he “knows” or “‘under- 
stands’’, or ‘‘can do”’. 


Feasible: It must be ensured that what 
the student is required to do can actually 
be done, within the time allowed and 
with the facilities to hand. Remember, 
too, the basic condition for feasibility: 
the minimum (practical, communication 
and intellectual skills) to qualify for the 
course. This is the prerequisite level. 


Logical: the objective must be internally 
consistent. 


Observable: it is obvious that unless there 
is some means of observing progress 
towards an objective, it will be impossible 
to tell whether the objective has been 
achieved. 


Measurable: One often hears “most of 
what I teach is intangible and cannot be 
measured’’. Even rough measurement is 
better than none at all, for if no measure- 
ment is made instructors tend to assume 
that a goal has been achieved just because 
they have taught the subject. If your 
teaching skills cannot be evaluated, you 
are in the awkward position of being 
unable to demonstrate that you are 
teaching anything at all. That is why the 
objective must include an indication of 
acceptable level of performance on the 
part of the student. 


This does not mean that an objective 
that does not lend itself to measurement 
by present-day techniques is necessarily 
a “bad” objective. On the other hand, 
the existence of a criterion for measure- 
ment will make it easier to choose or 
construct a valid evaluation mechanism, 
however sketchy this mechanism may be 
at the start. 
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Prerequisite Level 


What the learner has to be able to ‘do’ before 
undertaking an educational programme. 
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words often used words open to 
but open to many fewer interpretations 
interpretations 


to know to write 

to discuss to identify 

to understand to differentiate 

to really understand to solve 

to appreciate to construct 

to fully appreciate 16. 7NSe 

to believe to compare 

to have faith in to contrast 
etc. 


Mager, 1962 
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Abbreviate 
Act 
Administer 
Aid 

Allow for 
Analyse 
Apply 
Appraise 
Arrange 
Assist 

Ask 
Assemble 
Attend 
Audit 
Avoid 
Bring 
Build 
Calculate 
Care for 
Categorize 
Change 
Chart 
Check 
Choose 
Circle 

Cite 

Clean 
Close 
Collaborate 
Collect 


Communicate 


Compare 
Compile 
Complete 
Compute 
Conclude 
Conduct 
Connect 
Construct 
Contrast 
Contribute 
Control 
Convert 
Cooperate 
Correct 
Create 
Criticize 
Decide 
Decrease 
Deduce 
Defend 
Define 
Delimit 
Demonstrate 
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— nonrestrictive list — 


Derive 
Describe 
Design 
Designate 
Detect 
Determine 
Develop 
Diagnose 
Diagram 
Differentiate 
Direct 
Discover 
Discriminate 
Display 
Dissect 
Distinguish 
Divide 

Do 

Draw 

Edit 
Effect 
Encourage 
Enumerate 
Enunciate 
Establish 
Estimate 
Evaluate 
Examine 
Exchange 
Execute 
Explain 
Extract 
Extrapolate 
Facilitate 
Fill 

Find 
Follow 
Formulate 
Furnish 
Generalize 
Generate 
Get 

Give 
Guide 
Hold 
Identify 
Illustrate 
Implant 
Include 
Increase 
Indicate 
Induce 
Infer 
Inform 


Insert 
Integrate 
Isolate 
Justify 
Label 
Lead 
List 
Locate 
Maintain 
Make 
Manipulate 
Map 
Match 
Measure 
Meet 
Mobilize 
Modify 
Move 
Name 
Narrate 
Note 
Obtain 
Omit 
Operate 
Oppose 
Order 
Organize 
Outline 
Paraphrase 
Participate 
Perform 
Persuade 
Pick 
Place 
Plan 
Play 

Plot 
Point 
Position 
Practise 
Predict 
Prepare 
Present 
Prevent 
Prognose 
Promote 
Protect 
Provide 
Pursue 
Put 
Raise 
Read 
Rearrange 
Reassure 


List of active verbs for stating educational objectives 


Recite 
Reconstruct 
Record 
Recount 
Reduce 
Regroup 
Relate 
Remove 
Reorder 
Reorganize 
Rephrase 
Replace 
Request 
Reset 
Resolve 
Respond 
Restate 
Safeguard 
Select 
Send 
Separate 
Serve 

SCE 

Share 
Simplify 
Solve 
Sort 
Speak 
Specify 
Start 
State 
Store 
Structure 
Suggest 
Supply 
Support 
Synthesize 
Tabulate 
Take responsibility (for) 
Teach 

Tie 

Time 
Trace 
Translate 
Treat 
Underline 
Use 
Utilize 
Verify 
Wash 
Weigh 
Work 
Write 


If you feel the list on the previous page is incomplete, go ahead . . . . add the active verbs relating 
to the category of health personnel that interests you. 
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Relevance Is 
the essential quality 
of educational objectives 


Objectives which have every 


quality except relevance 


are potentially dangerous | 


elements of a specific 
educational objective 


the act 
the content the task 
the condition 


+ the criteria 
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To say what one means by an objective is neither to 
reduce the importance of the objective nor its pro- 
fundity . . . the act of writing it down means merely 
that what was once secret is now open for inspection 
and improvement. 

R.F. Mager 
Meee ote Gs es a ea a a 
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description of the elements of a 
specific educational objective 


1. Identify the act (for example, by underli- 


a Act and Content oe a: 
ning it in the sentence). 


_ The act is the description of the task 


aimed at, expressed by an active verb. 2. Decide whether is is an actual act or a 
2 E . . 
__ The content specifies the subject in re- performance indicator. 
lation to which the ee ‘ a be performed. 2.1 If it is a performance indicator, decide 
: roscope 
(Example: “Repair a binocular mic P whether it enables an inference to be made 


or “take” a sample of venous blood”. The 
act is the verb in italic; the rest of the sen- 
tence is the content.) 
It is perfectly acceptable for the description 
of the act (of a specific objective) to be stated 
in terms of a “performance indicator” and (b) If not, write another one. 
not in terms of the actual act required, as 2.2 If it is an actual act, decide whether it is 
long as the latter is quite clear. as eee ise 

sere ; aks plicit or implicit. 
The performance indicator is the description 
of an act whose satisfactory performance 
implies that the student is able to accomplish 
the actual required. (Example: “Underline” 


concerning the actual act. 


(a) If so, decide whether it can be simplified 
and whether it corresponds to the stu- 
dent’s level. 


(a) If it is explicit, decide whether it can be 
simplified and whether it corresponds 
to the student’s level. 


in the diagram the names of all the veins that (b) If it is implicit, include a “performance 
pass in front of the corresponding artery”.) indicator”... 

Here it is obvious that if the act consists of | 
“underlining” it is only a performance indi- a Condition. — This is the description of the 
cator which shows that the student is able to circumstances in which the act must take 
“distinguish veins passing in front from place (data, restrictions and limitations). 
those in any other position”. 

In all cases, the appropriate procedure with a Criterion. — The definition of the accepta- 
regard to the act (the actual act or a perfor- . ble level of performance expected from the 
mance indicator) is as follows: student. 
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Let us take a specific educational objective and identify 
each of its elements: 


“Identify on frontal X-ray films of the thorax the 
presence or absence ‘of opacities of the pulmonary 
parenchyma, of more than 2cm diameter in 80% of 
cases,” 


Act: “Identify the presence” indicates the act to be 
performed. 


Content: ‘‘Opacities of the pulmonary parenchyma.” 


Condition: ‘‘Frontal X-ray films of the thorax” are 
provided to the student. 


Criterion: The student must identify “in 80% of cases 
any opacity of over 2cm diameter’. 
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Some people mistakenly conclude that there is no 
difference between objectives and test items. That is 
true only in the case where the main intent of the 
objective is overt. 


R.F. Mager 
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You can check your choice by turning to p. 1.60, questions 15 — 20 and the answers on p. 1.62. 


Identify the four elements of the following objective: 


To be able to: repair a binocular microscope (brand X, Y or Z) having been informed of the 


defect and given a descriptive diagram, appropriate tools and spare parts, so that the microscope 
functions according to specifications. 


m Act: 
= Content: 
= Condition: 


® Criterion: 


Answers to exercise on page 1.41 


| a fatatals}el7{s}ol 10] 11] 12] 13 | 14 | 5 | 16 | 17 | 18 | 19 
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If your choices were different, it does not necessarily mean they are “wrong”. Possibly the active 
verbs used for some of the tasks are open to several interpretations. 
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Please Note! Before starting this exercise cover the area inside the dotted line 00 OO 000 


Decide which of the specific educational objectives given below (prepared by participants in 
workshops) conform to the standards described in the preceding pages. Is each one really a 
professional task? Does it include an act and content, conditions and a criterion? (Answer Yes 


or No). What is its principal component? 
Professional} Act and ry eee Principal 
task content | Condition | Criterion component 
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Specific educational objectives. 
The student must be able to: 


1. Make the diagnosis of 
anaemia on the basis of a detailed 

haematological picture described 

in the patient’s records. 


2. Determine the health conditions of 
a family in its environment by 
making three home visits. 


5 
zo |-| 
O 


OO 


DO. 


During a prenatal consultation, 
recognize at least three major 
symptoms of pre-eclampsia. 


Give oral care with the available 
equipment (according to technical 
sheet X) to a patient confined to 

bed and conscious. 


Or OG) 
Oxo 


Yes Yes Yes Yes + ++ 


Read one issue of a professional 
journal every fortnight, outside 
working hours. Criterion: write a 

summary (not more than 10 lines) 
of at least one of the articles. 
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During a prenatal consultation, carry 
out examinations for detecting and 
preventing complications of pregnancy 
(according to technical sheet Y). 


Yes 
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7. Make a survey (prepare questionnaires, 
carry out the survey, interpret the 
results) of the habits and customs of a 

population for the identification of 
those which represent a danger to health. 


Give from memory two similar and two 
dissimilar characteristics concerning the 
immunological value of and the 
epidemiological indications for 

(a) inactivated, and (b) attenuated 
polio vaccines. 


Measure the length of a newborn baby, 
using an infant measuring board, with 
a maximum error of 1 cm. 
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10. Identify by microscopic examination 
three of the following micro-organisms: 

meningococcus; Hansen’s bacillus; 

human tubercle bacillus; gonococcus. 


OLD Are O 


11. Using an optical microscope 
(magnification x 40), diagnose 
granulation tissue on a slide in five 
minutes, indicating at least five points 
of recognition present on the slide. 


OO «Gk O 


Yes Yes Yes 
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12, Make a macroscopic diagnosis of a 
benign breast tumour in an operation 
specimen and indicate at least four 


Yes 
es 
es 
es 

. . . = 
characteristics of benignity observed. 


ihe me i ee 6 
Y 
Y 


One O%s G) 


1.52 


Don't forget! 


The essential quality of 


educational objectives is 


their re/evance to the 


health needs of society 


The test of triviality of an educational objective is not in the words 
of that objective or in a statement of performance. It is in the 
consequence of not achieving the performance. 

Mager. In: Goal Analysis, p. 52 


1. Draw up specific educational objectives’ corresponding to three professional tasks that are 
familiar to you, stating explicitly what you feel the student should be able to ‘“‘do” in relation 


to the general and intermediate objectives. 


~ 


2. Identify the principal component of each objective (domain of practical, communication 
or intellectual skills). 


3. Define the criterion indicating the acceptable /evel of performance for each objective. 


The student must be able to perform the following tasks: 


1st task 


2nd task 


3rd task 


1 
f , 
If necessary, refer back to the section dealing with their essential qualities, pp. 1.35 — 1.50 
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4. For one of the tasks, describe the theoretical knowledge you think the student should have if 
he is to be capable of working towards the corresponding objective. 


Set out the content of this prerequisite level in the form of specific educational objectives. 
Check that they include all the necessary elements (act and content, conditions, criteria). 


To show that he has the knowledge needed for task no. .... , the student must be ab/e to: 


5. For one of the tasks describe features of behaviour showing that the student has the 
communication skills you would regard as adequate were he responsible for a member of 
your own family. Describe this behaviour in terms of specific objectives. 


For task no. ..., the student should have the attitude shown by the fact that he is able to: 


Check your Results 


Indicate whether each of the specific objectives you have defined above satisfies the following | 


conditions: 

(1) Is it really a professional task (or derived from one)? (2) Has the principal domain been 
identified? (3) Does it include the four elements: act, content, condition and criterion? (The 
criterion must relate to the principal component: communication skill (C), practical skill (P) 


or intellectual skill (I)). 


~ Content Condition 


Yes Yes 


No 


a 
(2) 
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Instructions: for each question select one answer. 


(Check your answers on p. 1.62) 


Question 1. 


Which of the following statements is in contradiction to the theories expounded in this chapter: 


A. The change in behaviour resulting from a learning activity is called performance. 

B. Final behaviour is the designation of the observable act which will be accepted as proof 
that the student has achieved a given educational objective. 
The definition of the objective of a course is a description or summary of the programme. 
The teaching intent explained by the educational objective indicates what should be the 
final behaviour of the student. 

E. None of the above statements. 

Question 2. 


An educational objective corresponds to all the following elements except one. Indicate which: 


A. It is sometimes also called a learning objective. 
B. It defines explicitly what the teacher should do. | 
C. It should be the basis for the preparation of the students’ timetable. 
D. It can be general or specific. 
: E. It is defined in behavioural terms corresponding to the tasks to be accomplished. 
Question 3. 


Mager stresses the need to define educational objectives in operational terms (description of the 
type of behaviour aimed at). Among the following qualities select the one which does not 
correspond to one of the advantages sought by this technique. 


A. 
B. 
C. 


It enables the student to evaluate his progress throughout his period of learning. 
It enables the instructor to choose relevant examination questions. 
It enables the instructor to make a satisfactory choice of teaching methods. 


It enables the student to acquire a thorough knowledge of the content, organization and 
timetable of a course. 


It enables the student to show clearly, at the end of the course, whether or not he has 
acquired specific types of behaviour. 


Question 4. : 
A specific educational objective has all the following qualities except one. Indicate which: 


It clearly communicates an educational intent. 


A. 

B. It makes objective evaluation possible. 

C. It defines a measurable behaviour. 

D. It establishes success criteria. 

E. It facilitates analysis of functions and tasks. 
Question 5. 


Indicate which of the following statements does not correspond to the principles of rational 
and effective educational planning: 


A. 
B. 


Institutional objectives should be defined before selecting a student evaluation system. 


Institutional objectives should be derived from the objectives of the various departments 
of a school. 


Regional health needs should be considered before drawing up an educational system for 
training members of the health team. 


Educational objectives should be defined before selecting educational methods and learning 
activities. 


National resources and budgetary constraints should be taken into account before defining 
general objectives. 


Questions 6 to 11. 


There have been numerous classifications of educational objectives. One of them considers 
three domains: . 


1. domain of communication skills. 
2. domain of practical skills. 
3. domain of intellectual skills. 


Using the following code: 


A= 1 D = 1 and 3 
B= 2 E = 2 and 3 
C= 3 


indicate the domain(s) to which the following objectives correspond: 


Question 6. 


The student should be able to name four new ideas concerning tuberculosis control. 


Question 7. 


The student should be able to measure the | 
of 1 cm. 
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ength of a newborn infant with a maximum error 


Question 8. 


The student should be able, when contraceptives are requested by an 18-year-old girl, to give 
information on four available methods without expressing any moral judgement. 


Question 9. 


The student should be able to avoid giving guilt feelings to a five-year-old enuretic child. 


Question 10. 


The student should be able to construct, using simple data already tabulated, a histogram 
including title, coordinates and additional details without any mistake in the curve. 


Question 11. 


The student should be able to make a health education poster for an anti-smoking campaign, 
given a model and the necessary materials (paper, felt, charcoal, coloured pencils, glue, scissors). 


Question 12. 
Indicate which of the following definitions corresponds to the prerequisite level: 
A. What the student should be able to do at the end of the curriculum. 


B. What the teacher should summarize before every course so as to place all the students on 
the same level. | 


The level reached by the student while pursuing an educational objective. 
What the student should be able to do before undertaking an educational programme. 


None of the above. 


Question 13. 
Which of the following statements best corresponds to the educational aim to be achieved: 


A. The student is perfectly familiar with the anatomical structure of the thorax. 
B. The student appreciates the approach centred on the basic needs of the patient. 


C. The student describes in writing the various steps of a nursing care plan in relation to the 
needs of a patient. 


D. The student has a thorough knowledge of the differences between a normal and a premature 
infant. 


E The student has a thorough understanding of professional ethics. 
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Question 14. | 
Indicate which of the following statements includes the four elements of an educational 


objective: 


A. 


The teacher will give five one-hour lectures to a group of 20 undergraduate students. The 
lectures will deal with the physiopathological mechanisms of the inflammatory process. 


During a simulation exercise, the student will give a brief written definition of the problem 
(at least 25 words) and list at least three alternative solutions, indicating which he would 
adopt and giving the reasons for his choice. | 


The student will show his knowledge of the mechanism of the action of aspirin on the 
subcortical cells. 


Using an optical microscope (magnification x 40), the student will diagnose granulation 
tissue in five minutes, indicating at least four diagnostic elements present on the slide. 


The student will demonstrate to his teacher, at the patient’s bedside, that he has a good 
grasp of clinical method and sound critical judgement, without any prejudice to his relation- 
ship to the patient. 


Questions 15 — 20. 


Instructions. The following statement comprises certain elements numbered 1 to 4. Using the 
code given below, select the element(s) that correspond to each of questions 15 — 20. 


(1) Repair (2) a binocular microscope (brand X, Y or Z) (3) having been informed of the defect 
and given a descriptive diagram (of brand X, Y or Z), appropriate tools and spare parts, (4) so 
that the microscope functions according to specifications. 
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Questions 15: Which is the task? 
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16: = Which is the act? 

17: Which is the specific educational objective? 
18: Which is the content? 

19: Which is the condition? 


20: Which is the criterion? 


If you help each learner 


to develop his/her learning 


objectives you may not 


have to do much else. 


adapted from Mager 


Suggested answers for the exercise on pages 1.57 — 1.60 


Suggested Answers i a ay did not give the right answer, reread the following 


C 


B 


D 


1.03 — 1.36 


1.36 — 1.41 


1.44 


1.43 — 1.50 


1.49 — 1.52 
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notice to the reader 


With some effort you have probably been 
able to reach most of the objectives indicated 
on page 1.02; this is the first and most 
important step but there is still a long way to 
go. Even to define all the specific objectives 
of your present teaching will be a long job. 
And after that you must make sure that they 
correspond to the general objectives, whether 
explicit or not . . . and it would be reassuring 
to know that they are really geared to the 
health problems of tomorrow’s population. 
Never forget relevance! And that is not all! 
Recent studies seem to show that teachers do 
not know how to use the educational 
objectives they have defined. There is no 
point in defining thousands of specific 
objectives if they are not then used as a 
reference for preparing the educational 
activities which will lead to their achievement. 


There are other awkward questions. Do the 
objectives you have defined really reflect 
what is important or merely what is relatively 
easy to set out in the form of objectives? 
Will they really help the students, and will the 
latter be better trained thanif ... etc. 


You will be faced with these arguments 
sooner or later, probably by those who make 
no attempt to define their own objectives or 
to learn how to use them. Whatever the limits 
and drawbacks of this approach it has the 
undeniable advantage of enabling studies 
to be made and research to be carried on. To 
find replies to the questions raised above and 
many others there is an urgent need for 
research whose scientific rigour will increase 
the credibility of the arguments, theories and 
hypotheses put forward by education 
specialists. For, however logical arguments 
may be, they must be backed up by some 
proof. Thus any attempt at dogmatism in 
this field or insistence on defining objectives 
at all costs should be avoided. These 
indispensable studies will call for considerable 
experimentation. 


You can help in carrying out such experi- 
ments. 


* 


If all this has not discouraged you, go on to 
the next chapter. Good luck! 
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It would be dangerous to measure with ever-increasing 
precision and objectivity educational objectives that are 
easy to measure (intellectual skills) and to neglect the 


more difficult ones (communication skills) 


: what must 


be measured are those that are important for the patient 


and the community. 
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evaluation 
planning 


evaluation planning 


ncepts in the field of educational evaluation. It stresses 


This second chapter presents basic co 


the very close relationship between evaluation and definition of educational objectives; and the 


primary role of any evaluation, w 


hich is to facilitate decision-making by those responsible for an 


educational system. It defines the subject, the purpose, the goals and the stages of evaluation and 


highlights the concepts of validity and relevance. 


Those who would like to learn more about these problems should consult the following 


publications: 


= Public Health Papers No. 52, WHO, “Development of educational programmes for the 


health professions’, 1974. 


= Evaluation of school performance, Educational documentation and information. 
Bulletin of the International Bureau of Education No. 184, third quarter 1972, 84 pages. 


After having studied this chapter and the reference documents mentioned you should be able to: 


1. Draw a diagram showing the relation- 
ship between evaluation and the other parts 
of the educational process. 


2. Define the principal role of evaluation, 
its purpose and its aims. 


3. Describe the difference between form- 
ative and certifying evaluation. 


4. List the good and bad features of a test. 


5 GOmpare the advantages and * dis- 


advantages of tests in current use. 


6. Define the following terms: validity, 
reliability, objectivity, and describe the 
relationship that exists between them. 


7. Choose an _ appropriate evaluation 
method (questionnaire, written examination, 
“objective” test (MCQ or short answer 
questions) or essay question, oral examin- 
ation, direct observation, etc.) for measuring 


2.02 


the students’ attainment of a specific 
educational objective. Compare the alter- 
natives in a specification table. 


8. Define (in the form of an organizational 
diagram) the organization of an evaluation 
system suitable for your establishment, and 
list the stages involved. 


Indicate: 


(a) the most important educational decisions 
you have to take; 


(b) the data to be collected to provide a 
basis for those decisions; 


(c) the aims of the system and sub-systems 


in terms of decisions to be taken and the 
object of each decision (teachers, 
students, programmes). 


9. Identify obstacles to and strategies for 
improvement of a system of evaluating 
students, teachers and programmes. 
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To change curricula or instructional methods without 
changing examinations would achieve nothing! 


Changing the examination system without changing 
the curriculum had a much more profound impact upon 
the nature of learning than changing the curriculum 
without altering the examination system 


G.E. Miller* 
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* International Medical Symposium No. 2. Rome. 23—26 March 1977 
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An analysis of educational innovations all 
over the world confirms G. Miller’s opinion. 
In this second chapter, therefore, you are 
invited to plan a system of evaluation that 
can be used as a basis for preparation and 
implementation of a programme. The process 
is already under way, for the formulation of 
specific educational objectives requires 
definition of criteria indicating the minimum 
level of performance expected from the 
stuuent. Educational decisions have to be 
made frequently during preparation and 
implementation of a programme; and the 
main purpose of evaluation is in fact to 
provide a basis for value judgments that 
permit better educational decision-making. 
First of all you must decide what you want to 
evaluate: students, teachers and/or pro- 
grammes. In each case you must determine 


what important educational decisions you 
will be expected to make in your capacity as 
teacher or administrator, for the instru- 
ments and mechanisms of evaluation pro- 
viding data for value judgments will be 
developed and used according to the type of 
decision required. A general methodology of 
evaluation and corresponding techniques 
do exist. Some are simple; others very 
complex and costly in time and money. Here 
again you will make your choice according 
to criteria that will ensure an adequate level 
of security. As in every educational process, 
you will have to shape all the consequences 
of your decisions into a coherent and logical 
whole. You are therefore invited to read the 
next pages before doing the exercise on 
D202. 
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The person who sets the examination controls the 


programme. 
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Education by objectives is not possible unless examin- 
ations are constructed to measure attainment of those 


objectives. 
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the educational spiral 


Defining 
Tasks and 
Educational Objectives 


Planning 


Implementing 


Evaluation an Evaluation . 


System 


Preparing and 


Implementing an 
Educational Programme 
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The evaluation process 
provides a basis for 
value judgments that 


permit better educational 


decision-making 


Notice to all teachers 


You are reminded that 
evaluation of education 


must begin with a clear 


and meaningful definition 


of its objectives 
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? 
Evaluation. .... 220s eee ee eee ee eee of whom: 


ig: OM: ENO Ree eee of what? 
Students 
© Teachers 
Programmes and courses 
NERS Venera in relation to what? 


O In relation to educational objectives. 


(They are the common denominator.) 
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Answer question 2 on p. 2.45. 
Check your answer on p. 2.48. 
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Before starting to define the organization, stages or methods of an evaluation system suitable 
for the establishment in which you are teaching, it would be useful to state: 


What important educational decisions* you think you and your colleagues will be taking over the 
next three years. 


* Examples of educational decisions: 


— to decide which students will be allowed to move up from the first to the second year 
— or to decide to purchase an overhead projector rather than a blackboard 
— or to decide to appoint Mr. X full professor. 


or to decide. 


You and your colleagues will have to make value judgments as a basis for each decision. It will 
therefore be useful to plan the construction and use of “instruments of evaluation” that will 
enable you to collect the data needed for making those value judgments 
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evaluation 
—a few assumptions’ 


Education is a process, the chief goal of which 
is to bring about changes in human behaviour 


m= The sorts of behavioural changes that the 
school attempts to bring about constitute its 
objectives 


= Evaluation consists of finding out the extent 
to which each and every one of these objectives 
has been attained, and determining the quality 
of the teaching techniques used and the 
teachers. 


* adapted from Downie. 


assumptions underlying basic 
educational measurement and 


evaluation’ 


Human behaviour is so complex that it cannot 
be described or summarized in a single score. 


The manner in which an individual organizes 
his behaviour patterns is an important aspect 
to be appraised. Information gathered as a 
result of measurement or evaluation activities 
must be interpreted as a part of the whole. 
Interpretation of small bits of behaviour as 
they stand alone is of little real meaning. 


The techniques of measurement and evaluation 
are not limited to the usual paper-and-pencil 
tests. Any bit of valid evidence that helps a 
professor or counsellor in better understanding 
a student and that leads to helping the student 
to understand himself better is to be con- 
sidered worthwhile. Attempts should be made 
to obtain all such evidence by any means 
that seem to work. 


The nature of the measurement and appraisal 
techniques used influences the type of 
learning that goes on in a classroom. If 
students are constantly evaluated on know- 
ledge of subject-matter content, they will 
tend to study this alone. Professors will also 
concentrate their teaching efforts upon this. 
A wide range of evaluation activities covering 
various objectives of a course will lead to 
varied learning and teaching experiences 
within a course. 


The development of any evaluation pro- 
gramme is the responsibility of the professors, 
the school administrators, and the students. 
Maximum value can be derived from the 
participation of all concerned. 


the philosophy of evaluation * 


1. Each individual should receive that 
education that most fully allows him to 
develop his potential. 


2. Each individual should be so placed that 
he contributes to society and receives per- 
sonal satisfaction in so doing. 


3. Fullest development of the individual 
requires recognition of his essential 
individuality along with some rational 
appraisal by himself and others. 


4. The judgments required in assessing an 
individual’s potential are complex in their 
composition, difficult to make, and filled 
with error. 


* Downie, N.M. Fundamentals of Measurement: Techniques 
and Practices. New York: Oxford University Press, 1967. 
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5. Such error can be reduced but never 
eliminated. Hence any evaluation can never 
be considered final. 


6. Composite assessment by a group of 
individuals is much less likely to be in error 
than assessment made by a single person. 


7. The efforts of a conscientious group of 
individuals to develop more reliable and 
valid appraisal methods lead to the clarific- 
ation of the criteria for judgment and reduce 
the error and resulting wrongs. 


8. Every form of appraisal will have critics, 
which is a spur to change and improvement. 


evaluation is 


a continuous process 


based upon criteria 


cooperatively developed 


concerned with measurement of 
the performance of learners, the 
effectiveness of teachers and the 
quality of the programme* 


* This chapter is mainly concerned with the evaluation of students. 
Evaluation of programmes and teachers is dealt with in chapter 4. 
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the psychology of e 


1. For evaluation activities to be most 
effective, they should consist of the best 
possible techniques, used in accordance with 
what we know to be the best and most 
effective psychological principles. 


2. For many years readiness has been 
recognized as a very important prerequisite 
for learning. A student is ready when he 
understands and accepts the values and 
objectives involved. 


3. It has long been known that people tend 
to carry on those activities which have success 
associated with their results. This has been 
known as Thorndike’s Law of Effect. 
Students in any classroom soon come to 
realize that certain types of behaviour are 
associated with success — in this case, high 
marks on a test or grades in a course. Thus, 
if a certain teacher uses tests that demand 
rote memory, the students will become 
memorizers. If a test, on the other hand, 
requires students to apply principles, interpret 
data, or solve problems, the students will 


valuation’ 


study with the idea of becoming best fitted 
to do well on these types of test items. In 
the long run, the type of evaluation device 
used determines, to a great extent, the type 
of learning activity in which students will 
engage in the classroom. 


4. Early experiments in human learning 
showed that individuals learn better when 
they are “constantly” appraised in a 
meaningful manner as to how well they are 
doing. 


5. The motivation of students is one of the 
most important — and sometimes the most 
difficult to handle — of all problems related 
to evaluation. It is redundant for us to say 
that a person’s performance on a test is 
directly related to his motivation. Research 
has shown that when a student is really 
motivated, performance is much closer to 
his top performance than when motivation 
is lacking. 


6. Learning is most efficient when there ts 
activity on the part of the learner. 
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Try to answer question 3 on p. 2.45. 


Check your answer on p. 2.48. 
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“Downie, N.M. Fundamentals of Measurement: Techniques 
and Practices. New York: Oxford University Press, 1967. 


2.14 


continuous evaluation 
formative and certifying evaluation 


Evaluation of education must begin with a 
clear and meaningful definition of its 
objectives. We cannot measure something 
unless we have first defined what it is we 
wish to measure. 


When this phase of evaluation (the definition 
of objectives) has been properly completed, 
the choice or development of suitable examin- 
ation procedures is that much easier. 
Schematically represented, the education 
cycle (p. 2.05) comprises the determination 
of objectives, the planning of an evaluation 
system, the development of teaching activities 
and the implementation of evaluation pro- 
cedures with possible revision of objectives. 


The role of evaluation should not be limited 
to one of penalization. It should not be just 
a series of only too frequent obstacles which 


the students are supposed to get over and 
which become their sole subject of concern, 
the actual instruction becoming quite 
secondary. Under these circumstances the 
student’s only interest is how to obtain his 
diploma with least effort. It is the teacher’s 
responsibility to convince the student that 
his education is directed towards wider 
aims than.merely gaining a diploma and that 
helping him to do so is not the sole purpose 
of evaluation (see p. 2.18). 


Evaluation should also be formative, 
providing the student with information 
on his progress. It must therefore be 
continuous. This concept has often been 
misinterpreted, resulting in constant harass- 
ment of the student. There is a fundamental 
difference between formative and certifying 
evaluation. 


You will find the following equivalents in the literature for these two expressions: 


Formative evaluation 


or 


diagnostic evaluation 


Certifying evaluation 
or 


summative evaluation 
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Continuous evaluation must pit the student against 
himself and his own lack of competence and not 


against other students. 
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formative evaluation® 


is designed to inform the student about 
the amount he still has to learn before 
achieving his educational objectives; 


measures the progress or gains made by 
the student from the moment he begins 
a programme until the time he completes 


its 


enables learning activities to be adjusted 
in accordance with progress made or 
lack of it; 


should in no way be used by the teacher 
to make a certifying judgment; the 
anonymity of the student should be 
safeguarded by use of a code of his 
choice. The coding system makes it 


possible to follow the progress of 
individuals and groups while preserving 
anonymity. 


is controlled in its use by the student 
(results should not appear in any official 
record); 


is very useful in guiding the student and 
prompting him to ask for help; 


is carried out frequently — as often as 
the student feels necessary; 


provides the teacher with qualitative and 
quantitative data for modification of 
his teaching (particularly contributory 
educational objectives) or otherwise. 


certifying evaluation 


is designed to protect society by pre- 
venting incompetent personnel from 
practising. 


is traditionally used for placing students 
in order of merit and justifying decisions 


as to whether they should move up to 
the next class or be awarded a diploma. 


— is carried out less frequently than form- 


ative evaluation: usually at the end of a 
unit or period of instruction. 
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Try to answer questions 4 — 8 on p. 2.45. 


Check your answers p. 2.48. 
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We don’t care how hard the student tried, we don’t 
care how close he got . . . until he can perform he must 
not be certified as being able to perform. 


R.F. Mager 
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* Read the article by C. McGuire — Diagnostic examinations 
in medical education. In; Public Health Papers No, 52. WHO 


Geneva, 1973. 
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Personal Notes 
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Strict Rule 
Formative evaluation should in no way be used by the teacher against the student. 
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student evaluation : what for 


9 Incentive to Learn (motivation) 
10 Feedback to student 


11. Modification of Learning Activities 


12 Selection of students appropriate 
measuring 
13 Success or failure techniques 


14 Feedback to teacher 
15 School public relations 


16 Protection of society 
(certification of competence) 


The numbers on the left refer to the exercise On the next page and 
the questions on pp, 2.45 — 2.47 
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aims of student evaluation’ 


1. To determine success or failure on the 
part of the student. This is the conventional 
role of examinations (certifying evaluation). 


2. To provide ‘feedback’ for the student: 
to keep him constantly informed about 
the instruction he is receiving; to tell him 
what level he has reached; and to make him 
aware through the questions of what parts 
of the course he has not understood (formative 
and certifying evaluation). 


3. To provide “‘feedback”’ for the teacher: 
to inform him whether a group of students 
has not understood what he has been trying 
to explain. This enables him to modify his 


teaching where necessary to.ensure that what 
he wishes to communicate to the students is 
correctly understood (formative and 
certifying evaluation). 


4. The “reputation of the school’ is some- 
thing of which the importance is not always 
evident, at least in European schools, whose 
reputation is often based not on an examin- 
ation system but on long-standing traditions. 
North American schools, on the other hand, 
customarily publish the percentage of 
students who have passed, for example, 
national examinations (formative and 
certifying evaluation). 
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NOW try IL: - 


indicate for each of the aims 


of evaluation (numbered 9—16 on the previous 
page) whether the measurement technique 
will be of the Certifying Evaluation type 


(C) or 


both Certifying and Formative 


Evaluation (CF). Check your answers on 


p. 2.48. 
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* Adapted from Downie, N.M. Fundamentals of Measure- 
ment: Techniques and Practices. New York: Oxford 
University Press, 1967. 
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four steps in student evaluation 


taking as one basis 


The criteria (acceptable level of performance) 
of the educational objectives 


Development and use of measuring instruments 


Interpretation of measurement data 


Formulation of judgments and taking of appropriate 
action 
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common methodology for 


student evaluation 


® Evaluation of practical skills 


® Evaluation of communication skills 


= Evaluation of knowledge and intellectual skills 


1. Make a list of observable types of 
behaviour showing that the objective pursued 
has been reached. 


2. Make a list of observable types of 
behaviour showing that the objective pursued 
has not been reached. 


3. Determine the essential features of 
behaviour in both lists. 


4. Assign a positive or negative weight to 
the items on both lists. 


5. Decide on the acceptable performance 
score. 


* For the last three stages obtain the agree- 


ment of several experts. 


Example. Objective: Reassure the mother of a child admitted to hospital 


Chew 3 [3 | ol 


Explain clearly 


often uses often uses 
what has been 
done to the 


child 


and rarely 
explains 
what they 
mean 


and never 
explains 
what they 
mean 


Minimum Performance 
Score: 


medical terms! medical terms 


etc. See the complete table on p. 4.24. 


The student should score n marks out of 10 on the rating scale. 


rarely uses rarely uses uses only 
medical terms} medical terms| terms suited 
and some- and always _ | to the 
times explains mother’s 
explains what they vocabulary 
what they mean 

mean 


exercise pooeeseeeecccccccces 


Try to answer questions 17 — 20 on p. 2.46 
and check your answers on p. 2.48. 
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See also Rezler, Public Health Papers No. 52, pp. 70-83, WHO, Geneva, 1973. 
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evaluation methodology according 
to domains to be evaluated 


Domain 


Intellectual Communication Practical 
Skills Skills Skills 


| 
| 


Carrying out 
of projects 


Written Tests 


“objective” type 


| 
essays aaa a 


“simulation” | 


— oral tests 
— observational rating scales 
— questionnaires 


Practical Tests 


in Real situations 
in Simulated conditions 
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For each of the educational objectives you have already defined (pp. 1.54, 1.55), choose from 
among the methods of evaluation set out on p.2.22 the one you think most suitable for informing 
you and the student* on the extent to which the objective has been achieved. 


Objective Method of evaluation Instrument of evaluation 


Indirect method 


Short, open-answer question based 
on the patient’s record 


1 page 1.52 


* For the purposes of this exercise the total number of students to be considered should be fixed: e.g., 100. 
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general remarks concerning 


examinations 


Analysis of the most commonly used tests 
shows that sometimes, often even, the 
questions set are ambiguous, unclear, dis- 
putable, esoteric or trivial. It is essential for 
anyone constructing an examination, whether 
of the traditional written type, an objective 
test or a practical test, to submit it to bis 
colleagues for criticism to make sure that 
its content is relevant (related to an 
educational objective) and of general interest, 
and does not exclusively concern a special 
interest or taste of the author; that the 
subject is interesting and real for the general 
practitioner or the physicians with a specialty 
different from that of the author; and that 
the questions (and the answers in the case 
of multiple-choice questions) are so formulated 
that experts can agree on the correct response. 
It is clear that a critical analysis along these 
lines would avoid the over-simplification of 


many tests which only too often justifies 
the conclusion: ‘“‘the more you know about 
a question the lower will be your score’’. 


The author of a test is not the best judge of 
its clarity, precision, relevance and interest. 
Critical review of the test by colleagues is 
consequently essential for its sound con- 
struction. 


Moreover, an examination must take the factor 
of practicability into account. This will be 
governed by the time necessary for its con- 
struction and administration, scoring and 
interpretation of the results, and by its general 
ease of use. 


If the examination methods employed become 
a burden on the teacher because of their 
impractical nature he will tend not to assign 
to the measuring instrument the importance 
it deserves. 


Piettueieee 6) OO oO ol Oo eo Oe) ot Bb er or 


A discussion is not always pertinent to the problem at 
hand, but one learns to allow for some rambling. It 
seems to help people realize that they normally use 
quite a few fuzzies during what they consider ‘‘technical 
discussions’’; it helps them realize that they don’t really 
know what they are talking about . . . a little rambling 
helps clear the air. Asking someone to define his goal 
in terms of performance is a little like asking someone 
to take his clothes off in public — if he hasn’t done it 
before, he may need time to get used to the idea. 


R.F. Mager 
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qualities of a test 


= Directly related to educational objectives 

= Realistic & practical 

= Concerned with important & useful matters 
= Comprehensive but brief 


= Precise & clear 


considerations of the type of 
competence a test purports to measure 


No test format (objective, essay or oral) 
has a monopoly on the measurement of the 
highest and more complex intellectual pro- 
cesses. Studies of various types of tests support 
the view that the essay and the oral examin- 
ation, as commonly employed, test pre- 
dominantly simple recall and, like the objective 
tests in current use, rarely require the student 
to engage in reasoning and problem-solving. 
In short, the form of a question does not 
determine the nature of the intellectual 
process required to answer it. 


Second, there is often a tendency to confuse 
the difficulty of a question with the com- 
plexity of the intellectual process measured 
by it. However, it should be noted that a 
question requiring simple recall may be very 
“difficult”? because of the esoteric nature 
of the information demanded: alternatively, 
a question requiring interpretation of data 
or application of principles could be quite 
“easy’’ because the principles of interpret- 
ation are so familiar and the data to be 
analysed so simple. In short, question difficulty 
and complexity of instructions are not 
necessarily related to the nature of the 
intellectual process being tested. 


Third, there is often a strong inclination to 
assume that any question which includes 


data about a specific case necessarily involves 
problem-solving, whereas, in fact, “data” are 
often merely “window dressing” when the 
question is really addressed to a general 
condition and can be answered equally well 
without reference to the data. Or, the data 
furnished about a “specific case” may con- 
stitute a “‘cut-and-dried’’, classical textbook 
picture that, for example, simply requires the 
student to recall symptoms associated with a 
specific diagnosis. It is interesting to note that 
questions of this type can readily be converted 
into problems that do require interpretation 
of data and evaluation, simply by making the 
case material conform more closely to the 
kind of reality that an actual case, rather 
than a textbook, presents. 


In short, just as each patient in the ward or 
out-patient department represents a unique 
configuration of findings that must be 
analysed, a test which purports to measure 
the student’s clinical judgment and _ his 
ability to solve clinical problems must simulate 
reality as closely as possible by presenting 
him with specific constellations of data that 
are in some respects unique and, in that sense, 
are new to him. Do not try to use a MCQ 
or a SOAQ to find out whether the student 
is able to communicate orally with a patient! 


Or .O.0)..2,0. 0) 0, Gove, bh Gall sD Dg po ile 1 


However reliable or objective a test may be, it is of no 
value if it does not measure ability to perform the 
tasks expected of a health worker in his/her professional 


capacity. 
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common defects of examinations 
(domain of intellectual skills) 


A review of examinations currently in use strongly suggests that the most common defects of 


testing are: 


triviality 


outright error 


ambiguity 


obsolescence 


bias 


complexity 


unintended cues 


the triviality of the questions asked, which is all the more serious in that 
examination questions can only represent a small sample of all those that 
could be asked. Consequently it is essential for each question to be 
important and useful; 


outright error in phrasing the question (or, in the case of multiple-choice 
questions, in phrasing the distractors and the correct response) ; 


ambiguity in the use of language which may lead the student to spend 
more time in trying to understand the question than in answering it; in 
addition to the risk of his giving an irrelevant answer; 


forcing the student to answer in terms of the outmoded ideas of the 
examiner, a bias which is well known and often aggravated by the teaching 
methods themselves (particularly traditional lectures); 


requesting the student to answer in terms of the personal preferences of 
the examiner when several equally correct options are available; 


complexity or ambiguity of the subject matter taught, so that the search 
for the correct answer is more difficult than was anticipated; 


unintended cues in the formulation of the questions that make the correct 
answer Obvious; this fault, which is often found in multiple-choice 
questions, is just as frequent in oral examinations. 


outside factors to be avoided 


In constructing an examination, outside factors must not be allowed to interfere with the factor 


to be measured. 


complicated instructions 
(ability to understand instructions) 


over-elaborate style 
(ability to use words) 


trap questions 
(ability to avoid traps) 


‘“test-wise’”’ 


In some tests, the instructions for students on how to 
solve the problems are so complicated that what is 
really evaluated is the students’ aptitude to under- 
stand the question rather than their actual know- 
ledge and ability to use it. This criticism is often 
made of multiple-choice examinations in which the 
instructions appear too complicated. The complexity 
is often more apparent than real and disturbs the 
teacher rather than the student. 


The student may disguise his lack of knowledge in 
such elegant prose that he succeeds in influencing the 
corrector, who judges the words and style rather than 
the student’s knowledge. 


This type of interference does not depend on a 
measuring instrument, but on possible sadistic 
tendencies on the part of the examiner who, during 
an examination, may allow himself to be influenced 
by the candidate’s appearance, sex, etc. Some 
candidates are more or less skilled at playing on 
these tendencies. 


This is a criticism that is generally made of multiple- 
choice examinations; it may in fact be applied to 
other forms of evaluation. In oral and written examin- 
ations, students develop a sixth sense, often based on 
statistical analysis of past questions, which enables 
them somehow to predict the questions that will be 
set. 
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comparison of advantages and 
disadvantages of different types of test 


Oral examinations 


Advantages Disadvantages 

1. Provide direct personal contact with 1. Lack standardization. 

candidates. 2. Lack objectivity and reproducibility 
2. Provide opportunity to take of results. 

mitigating circumstances into account. 3. Permit favouritism and possible abuse 
3. Provide flexibility in moving from of the personal contact. 

candidate’s strong points to weak areas. 4. Suffer from undue influence of 


4. Require the candidate to formulate irrelevant factors. 
his own replies without cues. 5. Suffer from shortage of trained 


5. Provide opportunity to question the examiners to administer the examin- 
candidate about how he arrived at an ation. 


ees 6. Are excessively costly in terms of 
6. Provide opportunity for simultaneous professional time in relation to the 
assessment by two examiners. limited value of the information it 


yields. 
Unfortunately all these advantages are 


rarely used in practice. 


Practical examinations 


Advantages Disadvantages 


1. Provide opportunity to test in a 1. Lack standardized conditions in lab- 


realistic setting skills involving all the oratory experiments using animals, in 
senses while the examiner observes and surveys in the community or in bedside 
checks performance. examinations with patients of varying 
2. Provide opportunity to confront the degrees of cooperativeness* 

candidate with problems he has not 2. Lack objectivity and suffer from 
met before both in the laboratory and intrusion or irrelevant factors. 


at the bedside, to test his investigative 
ability as opposed to his ability to apply 
ready-made “recipes”. 


3. Are of limited feasibility for large 
groups. 


4. Entail difficulties in arranging for 
examiners to observe candidates demon- 
strating the skills to be tested. 


3. Provide opportunity to observe and 
test attitudes and responsiveness to a 
complex situation (videotape recording). 


4. Provide opportunity to test the 
ability to communicate under pressure, 


to discriminate between important and “Standardize practical tests can be constructed; 


pea) issues, ‘6 arrange dhe data ink see “Simulation and evaluation in medecine” in 


Public Health Papers No. 61, pp. 18-34, WHO, 
final form. Geneva, 1974. = 
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Essay examinations 


Advantages 


1. Provide candidate with opportunity to 
demonstrate his knowledge and his ability 
to organize ideas and express them 
effectively. 


Multiple-choice questions 


Advantages 


1. Ensure objectivity, reliability and 
validity; preparation of questions with 
colleagues provides constructive criticism. 


2. Increase significantly the range and 
variety of facts that can be sampled in a 
given time. 


3. Provide precise and unambiguous 
measurement of the higher intellectual 
processes. 


4. Provide detailed feedback for both 
student and teachers. 


5. Are easy and rapid to score. 


Disadvantages 


1. Limit severely the area of the student’s 
total work that can be sampled. 


2. Lack objectivity. 
3. Provide little useful feedback. 


4. Take a long time to score. 


Disdavantages 


1. Take a long time to construct in order 
to avoid arbitrary and ambiguous questions. 


2. Also require careful preparation to 
avoid preponderance of questions testing 
only recall. 


3. Provide cues that do not exist in 
practice. 


4. Are ‘‘costly’” where number of students 
is small. 
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It is a highly questionable practice to label someone as 
having achieved a goal when you don’t even know what 
you would take as evidence of achievement. 


R.F. Mager 
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evaluation in education 


qualities of a measuring instrument 


a 


Some definitions 


1.1 


1.2 


Wh, 


1.4 


US 


1.6 


It should be recalled once more that 
education is defined as a process 
developed for bringing about changes 
in the student’s behaviour. At the 
end of a given learning period there 
should be a greater probability 
that types of behaviour regarded as 
desirable will appear; other types of 
behaviour regarded as undesirable 
should disappear. 


The educational objectives define the 
desired types of behaviour taken as 
a whole; the teacher should provide 
a suitable environment for the 
student’s acquisition of them. 


Evaluation in education is a 
systematic process which enables the 
extent to which the student has 
attained the educational objective 
to be “measured”. Evaluation always 
includes measurements (quantitative 
or qualitative) plus a value judgment. 


To make measurements, measuring 
instruments must be available which 
satisfy certain requirements so that 
the results mean something to the 
teacher himself, the school, the 
student and society which, in the 
last analysis, has set up the educ- 
ational structure. 


In education, measuring instruments 
are generally referred to as “‘tests’’. 


Among the qualities of a test, what- 
ever its nature, four are essential, 
namely, validity, reliability, olject- 
wity and practicability. Others are 
also important, but they contribute 
in some degree to the qualities of 
validity and reliability. 


2. Qualities of a measuring instrument 


The four main qualities of any measuring 
instrument (examination) are: validity, 
reliability, objectivity and practicability. 


2id 


Validity: the extent to which the test 
used really measures what it is inten- 
ded to measure. No outside factors 
should be allowed to interfere with 
the manner in which the evaluation is 
carried out. For instance, in 
measuring the ability to synthesize, 
other factors such as style should 
not compete with the element to be 
measured so that what is finally 
measured is style rather than the 
ability to synthesize. 


The notion of validity is a very 
relative one. It implies a concept of 
degree, i.e., one may speak of very 
valid, moderately valid or not very 
valid results. 


The concept of validity is always 
specific for a particular subject. 
For example, results of a test on 
public health administration may be 
of very high validity for identific- 
ation of the needs of the country 
and of little validity for a cost/ 
benefit or cost/efficiency analysis. 


Content validity is determined by 
the following question: will this test 
measure, or has it measured, the 
matter and the behaviour that it 
is intended to measure? 


Predictive validity is determined by 
questions such as the following when 
the results of a test are to be used 
for predicting the performance of a 
student in another domain or in 
another situation: 


To what extent do the results 
obtained in physiology help to 
predict performance in pathology? 

To what extent do the results 
obtained during the pre-clinical years 
help in predicting the success of 
students during the clinical years? 


2.2 Reliability: this is the consistency 
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with which an instrument measures 
a given variable. 


Reliability is always connected with 
a particular type of consistency: 
the consistency of the results in 
time; consistency of results according 
to the questions; consistency of the 
results according to the examiners. 


Reliability is a necessary but not a 
sufficient condition for validity. In 
other words, valid results are 
necessarily reliable, but reliable 
results are not necessarily valid. 
Consequently, results which are not 
very reliable affect the degree of 
validity. Unlike validity, reliability 
is a strictly statistical concept and 
is expressed by means of a reliability 
coefficient or through the standard 
error of the measurements made. 


Reliability can therefore be defined 
as the degree of confidence which 
can be placed in the results of an 
examination. It is the consistency 
with which a test gives the results 
expected. 


Objectivity: this is the extent to 
which independent and competent 
examiners agree on what constitutes 
a ‘“‘good”’ answer for each of the 
items of a measuring instrument. 


Practicability depends upon the time 
required to construct an examination, 
to administer and score it and to 
interpret the results, and on_ its 
overall simplicity of use. It should 
never take precedence over the 
validity of the test. 


3. Other qualities of a measuring instrument 


ona 


*This definition of efficiency has a narrower meanin 
instruments (pp. 2.33-2.37). 
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Relevance: this is the degree to 
which the criteria established for 
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3.6 


3.7 


3.8 


selecting questions (items) so that 
they conform to the aims of the 
measuring instrument are respected. 
This notion is almost identical to 
the one of content validity; and the 
two qualities are established in a 
similar manner. 


Equilibrium: achievement of the 
correct proportion among questions 
allocated to each of the objectives. 


Equity: extent to which the questions 
set in the examination correspond to 
the teaching content. 

Specificity: quality of a measuring 
instrument whereby an_ intelligent 
student who has not followed the 
teaching on the basis of which the 
instrument has been constructed will 
obtain a result equivalent to that 
expected by pure chance. 


Discrimination: quality of each 
element of a measuring instrument 
which makes it possible to dis- 
tinguish between good and poor 
students in relation to a _ given 
variable. 


Efficiency: quality of a measuring 
instrument which ensures the greatest 
possible number of independent 
answers per unit of time.* 


Time: it is well known that a 
measuring instrument will be less 
reliable if it leads to the introduction 
of non-relevant factors (guessing, 
taking risks or chances etc.) because 
the time allowed is too short. 


Length: the reliability of a measuring 
instrument can be increased almost 
indefinitely (Spearman-Brown for- 
mula) by the addition of new 
questions equivalent to those con- 
stituting the original instrument. 


g than the one given in the glossary (p. 6.05); it applies only to evaluation 


validity 


reliability 


objectivity 


practicability 


The extent to which the instrument 
measures what it is intended to 
measure. 


The consistency with which an 
Instrument measures a_ given 
variable. 


The extent to which independent 
and competent examiners agree on 
what constitutes a good answer 
for each of the elements of a 
measuring instrument. 


The overall simplicity of use of 
a test, both for test constructor 
and for students. 
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relationships between the . 
characteristics of an examination 


The diagram on the next page, suggested by 
G. Cormier, represents an attempt to sum up 
the concepts of testing worked out by a 
number of authors. However, no diagram 
can give a perfect representation of reality 
and the purpose of the following lines is to 
explain rather than justify the diagram. 


A very good treatment of all these concepts 
will be found in the book by Robert Ebel 
entitled Measuring Educational Achievement 
(Prentice Hall, 1965). 


Validity and reliability 


Ebel shows that ‘“‘to be valid a measuring 
instrument (test) must be both relevant and 
reliable.’’ This assertion justifies the initial 
dichotomy of the diagram. It is, moreover, 
generally agreed that ‘‘a test can often, if 
not always, be made more valid if its reliability 
is increased.” 


Validity and relevance 


According to Ebel’s comments, it seems that 
the concept of relevance corresponds more 
or less to that of validity of content. In any 
case, both are established in a similar manner 
(by consensus). 


By definition, a question is relevant if it adds 
to the validity of the instrument, and an 
instrument is relevant if it respects the specific- 
ations (objectives and taxonomic levels) 
established during its preparation. 


Relevance and equilibrium 


It seems, moreover, that the concept of 
equilibrium is only a sub-category of the 
concept of relevance and that is why the 
diagram shows it as such. 


Relevance and equity 


It seems evident that if the instrument is 
constructed on the basis of a content itself 
determined by objectives, then it will be 
relevant by definition. If this is not done, 
then the instrument will not be relevant and 
consequently not valid. It is equitable in the 
first case and non-equitable in the second. 
However, an examination can be equitable 
without being relevant (or valid) when, 
although it corresponds well to the teaching 
content, the latter is not adequately derived 
from the objectives. 


Equity, specificity and reliability 

The diagram reflects the following implicit 
relationship: a test cannot be equitable if 
it is not first specific. Specificity, just like 
equity and for similar reasons, will affect the 
reliability of the results. 


Reliability, discrimination, length, bomo- 
geneity (of questions) and heterogeneity 
(of students) 


According to Ebel, reliability is influenced by 
the extent to which the questions (items) 
clearly — distinguish competent from 
incompetent students, the number of items, 
the similarity of the items as regards their 
power to measure a given skill and the extent 
to which students are dissimilar with respect 
to that skill. The discriminating power of a 
question is directly influenced (see pages 
4.60 — 4.61) by its level of difficulty. The mean 
discrimination index of an instrument will 
also be affected by the homogeneity of the 
questions and the heterogeneity of the 
students. From the comments made above it 
can be seen how equity and specificity will 
also influence the discriminating power of the 
instrument, 
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Try to answer questions 22 — 25 on p. 2.47 
and check your answers on p. 2.48. 
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relationships between 
characteristics of an examination* 


Validity 
Reliability Relevance 
Objectivity Discrimination Equilibrium 
Equity 
Say Homogeneity 
(exam.) 
Specificity 


Heterogeneity 


(students) 
Efficiency 
Duration Length 
(Time) | (number of Q) 


* as proposed by G. Cormier, Universite Laval, Quebec. 


N.B. Additional relationships to those suggested in this diagram can be established. The number 
of arrows has been kept to a minimum for the sake of clarity and to give a basic idea of the 


concept as a whole. 


For each of the educational objectives you defined on page 1.54, describe two methods of 


evaluation that seem suitable to you for informing yourself and the student on the extent to 
which that objective has been achieved. Compare the two methods on the basis of the three 


criteria shown in the table below. 


Make a differential diagnosis of 


anaemia based on the detailed ieee FRAN ae ee 
haematological picture described Validity — Objectivity Practicability 


in the patient’s medical record. 


Modified essay question. A series 
of 10 short questions based on 
patient’s record as supplied to 
student (1 hour). 


Student given patient’s record 
(10 mins.) followed by 15 min. 
oral examination. 


Methods of Evaluation for a class of 200 students 


The student should be able to: 


Validity Objectivity Practicability 


Check the meaning of the words validity, objectivity and practicability in the glossary page 6.01 
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For evaluation 
the essential quality is validity 


but don’t forget 


that for an educational 


system considered as a whole 


it is its relevance 
that is of primary importance 


evaluation is a matter for teamwork 


The planning of an evaluation system is 
obviously not simple. It is a serious matter, 
for the quality of health care will partly 
depend on it. It has been stressed many times, 
moreover, that it should be a group activity. 
We have stated in the preceding pages that 
evaluation must be planned jointly; that 
implementation of any evaluation programme 
is the responsibility of the teachers, in 
collaboration with students and_ the 
administration; that evaluation carried out 
jointly by a group of teachers is less likely 
to be erroneous than when carried out by 
one person; and, finally, that critical analysis 
of a test by colleagues is essential to its sound 
construction. 


This work performed jointly by a group 
of teachers calls for a coordinating mechanism. 
The terms of reference of each group and 
group member must be defined explicitly 
and known to all. The institution’s higher 
authorities must provide the working groups 
and their members with the powers corres- 
ponding to the task to be accomplished. 


The diagram on the next page shows one 
type of organization and meets the needs of 
a given institution. Other types of organization 
can be envisaged, according to existing 
structures and local traditions. 
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Do not change anything that works satisfactorily .. . . 
what is satisfactory to some, however, is not necessarily 
good enough for others. Teaching is a matter for team- 


work. 
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Organizational diagram showing relationships between curriculum committee, evaluation 
committee and teaching units 


School Administration 
Transmits Institutional 
Objectives 


Educational Consultant 


Evaluation Committee — 


Prepares Comprehensive 
Test 


Curriculum Committee 
Prepares Specification 
Tables 


The second year The Nth year 
coordinator transmits the 


intermediate objectives 


The first year 
coordinator transmits the 
intermediate objectives 


coordinator transmits the 
intermediate objectives 


Each unit prepares the specific objectives and the test items 
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Personal Notes 
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Judge the consequence of the student's not achieving 
the objective by answering such questions as: “If he 
cannot perform the objective when he leaves my 
instruction he is likely to... . . “’. The answer should 
help you decide how much energy to put into 
constructing a valid evaluation system to find out 
whether the objective is achieved as written, 


R.F. Mager 
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Show graphically the type of organization (commissions, committees, administrations, etc., 
with a description of their functions) which seem desirable (in the establishment where you are 
teaching) for introducing (or improving) an evaluation system capable of providing the necessary 
data for checking that all educational objectives have been achieved (do not forget also to take 
account of the decisions you have formulated on page 2.09); then compare with page 2.41. 
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Describe the obstacles you are liable to encounter in applying the organizational plan you have 
imagined on the previous page and indicate tactics for overcoming each of these obstacles. 


Obstacles Tactics 
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et ee 


(Check your answers on p. 2.48) 


Question 1 
The main role of evaluation is: 


Question 2 
The purpose of evaluation is to make a value judgment concerning: 
A. Students and programmes. 


B. Students and teachers. 

C. Programmes and teachers. 
D. Students. 

E. None of the above. 
Question 3 


Thorndike’s ‘‘Law of Effect’’ is based on the fact that: 
A. Students learn better when they are motivated. 
B. Students learn better when they play an active role. 


C. Students are receptive when they understand the educational objectives which have been 
defined. 


D. Students tend to engage in activities which have success associated with their results. 
E. Students work better if the teacher makes an impression on them. 


Questions 4 to 8 
Indicate to which of the following each question refers: 
A. Formative evaluation ; 


B. Certifying evaluation 
C. Both 

D. Neither 

Question 4 


Its main aim is to inform the student on his/her progress. 


Question 5 
Does not preserve anonymity. 


Question 6 
Enables the teacher to decide to replace one programme by another. 


Question 7 
Justifies the decision to let a student move up from the second to the third year. 


Question 8 
Permits rank-ordering of students. 
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Questions 9 to 16 cop 
ay each of the aims of student evaluation (list numbered 9 to 16, p. 2.18), indicate whether the 


appropriate measuring instrument will be of the certifying evaluation type (C) or both certifying 


and formative evaluation (CF). 


Question 17 | 
The four steps of the process of student evaluation are as follows: 


if. 


en 
Si 
4 


Question 18 
All the following steps except one are essential in constructing any measuring instrument. 


A. Precise definition of all aspects of the type of competence to be measured. 
B. Obtaining reliability and validity indices for the proposed instrument. 


C. Making sure that the type of instrument chosen corresponds to the type of competence to 
be measured. 


D. Making sure, by an explicit description of the acceptable level of performance, that the 
use of the measuring instrument will ensure objectivity. 


E. Determination of the particular behaviour expected from individuals who have or have not 
acquired the specified competence. 


Question 19 


When evaluating communication skills (domain of interpersonal relationships), all the following 
steps should be taken except one: 7 


A. Describe specific types of behaviour showing a given affective level. 
B. Describe explicit types of behaviour showing the absence of a given affective level. 


C. Observe students in real situations enabling them to manifest the types of behaviour 
envisaged. 


D. Obtain the agreement of a group of experts on the relationship between explicit types of 
behaviour and the affective level envisaged. 


Obtain the student’s opinions on the way in which they would behave in specific situations. 


Question 20 


The essential variable to be considered in evaluating the results of teaching is: 
A. The student's performance. 


B. The opinion of the teacher and his colleagues. 

C. The opinion of the student regarding his performance. 
D. The satisfaction of the teacher and the students. 

E. The teacher's performance. 
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Question 21 

Which of the following is not suitable for measurement by written examinations of the 
“objective’’ type: 

Ability to recall precise facts. 

Ability to solve problems. 

Ability to make decisions. 

Ability to communicate with the patient. 

Ability to interpret data. 
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Questions 22 and 23. 
If the following qualities can be attributed to an examination: 


A.= Validity B = Objectivity C = Reliability D = Specificity E = Relevance 


Question 22 
What quality is obtained if a group of experts agree on what constitute good answers to a test? 


Question 23 
What quality implies that a test consistently measures the same thing? 


Question 24 
The following factors, except one, generally affect the reliability of a test: 


A. its objectivity. 

B. The mean discrimination index of the test questions. 
C. The homogeneity of the test. 

D. The relevance of the test questions. 

E. The number of questions in the test. 

Question 25 

Which of the following test criteria is influenced by all the others? 
A. Reliability. 

B. Validity. 

C. Objectivity. 

D. Specificity. 

E. Relevance. 


2.47 


Suggested answers for the exercise on pages 2.45 — 2.47 


If you did not find the correct answer, consult the 
Suggested Answer following pages again 


2.02 — 2.06 
2.08 
2.14 
2.15 — 2.19 
2.20 
rei 
2.07 — 2.35 
2.08 — 2.21 
2.27 — 2.31 
2.34 — 2.35 
2.34 — 2.37 
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programme 
construction 


the teaching-learning concept 
and programme construction 


This third chapter deals with the planning of educational programmes and learning Spee. i 
attempts to bring out the change which is now taking place, i.e. the tendency to be interest 

more in what the student receives, perceives and assimilates than in what the teacher presents, 
gives or does. It shows that this change can be of benefit to all (the teacher continuing to learn 


and the student taking over part of the teacher’s role). 


It deals with the teaching-learning process as part of a teaching-learning system. It stresses the 
effectiveness of methods which place the student in an active situation. It proposes a relevant and 
student-orientated plan of action taking into account the (traditional) obstacles to change. 


Those with a deeper interest in these problems are strongly advised to consult the following 
publications: 


= Public Health Paper No. 52,WHO, ‘‘Devel- m™ Technical Report Series No. 489, WHO, 


opment of educational programmes for the “Implications of individual and small group 
health professions’’, 1973 learning systems in medical education”, 
1972 


= Public Health Paper No.47,WHO, “‘Aspects m Technical Report Series No. 746, WHO, 
of medical education in developing coun- “Community-based education of health 
tries”, 1972 personnel”, 1987 


m Technical Report Series No. 521, WHO, 


m Public Health Paper No. 61, WHO, “Educ- “Training and preparation of teachers in 
ational strategies for the health pro- schools of medicine and of allied health 


fessions’’, 1974 sciences”’, 1973 


After having studied this chapter and the references mentioned, you should be able to: 


1. Explain the differences between ‘‘edu- 7. Indicate the steps involved in construct- 
cation’’, “teaching” and “‘learning”’, and ing a programme. 
describe the new trends in the teaching/ 
learning system and the various learning 8. Construct a programme or decide 
situations. whether a programme or course needs 


revision, using a specification table. 
2. Define the concept of relevance of a ore 


Fegeeen 9. Construct a self-learning package. 


3. List 10 conditions which facilitate 

learning and evaluate a learning activity. 10. Define the role which, as a teacher, you 
would like to assume in order to motivate 
and facilitate the learning of students 
for which you are responsible. 


4. Indicate the aims and general methods of 
teaching. 


5. Specify at least two advantages and two 


disadvantages for each technique and ake JUCGETy oe obstacles papi! bea keene? 

medium used in teaching countered im setting up a competency 

based curriculum geared to the health 

6. Select a teaching method that will make needs of the community, and describe 
an educational objective easier to attain. strategies for overcoming them. 


Compare the alternatives in a specifi- 
cation table. 
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the educational spiral 


Defining 
Tasks and 


Educational Objectives 


Planning 


Implementing 


Evaluation an Evaluation 


System 


Preparing and 


Implementing an 
Educational Programme 
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It was felt useful at this mid-point of the 
Educational Handbook to attempt to summarize 
the major concepts presented so far and place in 
perspective the two following chapters. We felt 
the following article would very well serve this 
purpose. | 


You will find later the name of the author and, 
what is more significant, when it was written. 
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distinctive attributes of 
education for the professions 


Educational ferment characterises schools and 
colleges throughout the world. New responsi- 
bilities, new definitions of educational aims 
and new programmes are emerging at all levels 
of the educational ladder. At first glance, these 
new developments appear to have no common 
elements but to be highly individual, unrelated 
incidents of educational change. 


It is helpful to consider first the essential 
characteristics of a profession and, second,the 
major tasks involved in planning and con- 
ducting an educational programme. In the light 
of these two factors, the significant elements 
of effective education for the professions can 
be seen more clearly. 


A profession based on a code of ethics 


From the standpoint of the education required, 
there are two essential characteristics of a true 
profession. The first is the existence of a 
recognised code of ethics. This ethical code 
commits the members of the profession to 
certain social values above the selfish ones of 
income, power, and prestige. In the case of 
medicine, for example, its code of ethics 
dedicates the doctor to the saving of lives and 
the protection of the patient’s health above 
all material and personal considerations. 


A professional code of ethics not only pro- 
fesses social values that are above selfish ones, 
but it expects the individual member seriously 
to dedicate himself to these higher values. 
Furthermore, a profession establishes some 
form of group discipline in support of these 
values. A doctor who violates the ethical code 
of medicine receives the disapproval of fellow 
doctors and is subject to discipline by the state 
for malpractice. One of the indications that an 
occupation is becoming a profession is a 
concerted movement among members of the 
occupation to establish and maintain group 
discipline in order to uphold the ethical 
values to which the group gives lip service. 


A common problem in several professions is 
to distinguish ethical values that are broadly 
social from a pseudo-ethical code that in 
reality dedicates the individual to the selfish 
interests of the group rather than to mankind 


generally. In some programmes for the edu- 
cation of doctors, there has been a tendency 
to develop a code of ethics in which the doctor 
dedicates himself to the medical profession 
and to the interests of doctors rather than to 
the improvement of the health of mankind. 


A profession is not a union. Professional 
ethics must focus upon higher social values 
than dedication to the selfish interests of the 
professional group. Yet to attain this high 
alm. is one of the serious problems of the 
professions. 


A profession based on principles 


The second distinguishing feature of a pro- 
fession is the basing of its techniques of 
Operation upon principles rather than rule- 
ofthumb procedures or simple routine 
skills. For an occupation to be a profession 
it should involve complex tasks which are 
performed by artistic application of major 
principles and concepts rather than by routine 
operations or skills. This is an important 
differentiating feature. A_ skilled trade, 
obviously, involves some fairly complex tasks, 
but the members of the trade are able to 
perform these tasks through acquiring certain 
routine skills and through following certain 
specified rules. Many of the problems en- 
countered by a member of a profession are in 
a certain sense unique. To solve such a 
problem he must draw upon certain basic 
principles. However, the application of these 
principles necessitates an analysis of the parti- 
cular problem to see what are its unique aspects 
which will require adaptation of the principles. 
This adaptation is an artistic task; that is, it 
involves individual judgement and imagination 
as well as skill. A skilled trade does not demand 
this type of intellectual operation. 


In the early days, surgery was not really a 
profession but was a skilled trade. Certain 
skills, such as those used in bone-setting, 
were transmitted from one generation of 
surgeons to another, and the surgeon learned 
largely as an apprentice how to carry on his 
trade. With the development of the basic 
medical sciences like anatomy and physiology, 
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it became possible to gain a more fundamental 
understanding of what was involved in bone 
structure and in tissue development, so that 
a surgeon with adequate scientific background 
was able to adapt his particular procedures to 
the specific conditions surrounding a given 
case. He then solved the problem in each case 
in terms of basic principles rather than simply 
following rule-of-thumb procedures. 


Whenever a member of any profession meets 
his day-by-day tasks in terms of routine 
performance, for him the occupation is no 
longer a profession. 


A profession not only utilises basic principles 
rather than depending on rule-of-thumb pro- 
cedures, but as it becomes more mature it 
recognises that the principles used in the 
profession must be viewed in an increasingly 
larger context and that, correspondingly, the 
science needed by the profession must be 
continually extended to more basic content 
rather than restricted only to the obvious 
appled science. Thus, increasingly has medi- 
cine come to recognise the interrelationship 
of nutrition, physiology, anatomy, bio- 
chemistry, and other fundamental sciences 
which give a much broader basis for under- 
Standing a particular medical condition of a 
given patient. 


In general it can be said that as a profession 
becomes increasingly mature, it not only 
develops members who carry on their work 
through principles rather than rule-of-thumb 
procedures, but it also encourages members 
to gain an understanding of these principles 
in a much larger context than that afforded 
by the usual confines of the occupation. 


I have emphasised these two major character- 
istics of a profession — the development of 
a code of ethics, and the use of techniques 
that are based on principles — both because 
they are the most significant differentia of 
professions from other occupations and 
because they help to indicate some of the 
fundamental tasks of professional education. 


Tasks in planning and conducting an edu- 
cational programme 


Shifting our attention from the essential 
characteristics of a profession to the tasks 
involved in planning and conducting an 
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educational programme, we find it possible 
to classify these tasks into four major types, 


namely: 


m deciding on the objectives; 


m selecting learning experiences that will 
contribute to the objectives; . 


m organising the learning experiences to 
maximise their cumulative effect; and 


= evaluating the effectiveness of the edu- 
cational programme in attaining its 
objectives through appraising the edu- 
cational progress of the students. 


Education is a process for changing the 
behaviour of students in desired dtrections. 
The term “behaviour’’ is used in the broad 
sense and includes thinking, feeling, and 
acting. When a student is educated he has 
acquired ideas, habits, attitudes, interests, 
ways of thinking, and professional skills 
which he did not have before he went to 
school; his behaviour has changed. 


From this definition of education it is clear 
that the educational objectives are the be- 
haviour patterns that the school tries to 
develop in the student. The knowledge, the 
skills, and the ways of thinking that the 
student is expected to acquire are examples 
of these objectives. The task of deciding on 
objectives is an important one because these 
are the aims and purposes which can and 
should be used to guide the entire educational 
programme. The only rational basis for select- 
ing learning experiences and devising eval- 
uation procedures is in terms of their relation 
to the educational objectives. 


The second major task of education is to 
select learning experiences that will con- 
tribute to the objectives. This raises a previous 
question: how do people acquire these desired 
changes in behaviour patterns? “People acquire 
them by practising them’’ is a simple but fairly 
accurate answer. Getting students to practise 
the desired behaviour, however, is not the 
easiest thing in the world. 


A student develops understanding by recalling 
ideas, by explaining them in his own words, 
and by finding illustrations of them. Skill in 
ways of thinking are developed by practising 
problem-solving again and again. Manual 


skills and habits are also acquired by practice. 
An attitude is acquired as the student looks 
repeatedly at the phenomenon from a new 
perspective. Interests are acquired by getting 
satisfaction from certain kinds of experiences 
so that the experiences become increasingly 
satisfying. For all of these kinds of behaviour, 
students acquire new behaviour patterns by 
practising them. 


One fact clearly emerges from this analysis — 
the teacher cannot learn for the student. 
Whether or not the student develops an 
understanding will depend upon what is 
going on in his mind, not what is going on 
in the teacher’s mind. To plan learning 
experiences is to outline the activities that 
will give the students a chance to practise 
the behaviour implied by the objectives. 
Thus, planning a particular course will mean 
providing situations in which students will 
encounter problems to solve so that they 
can gain understanding and develop critical 
thinking. Planning may also involve providing 
tasks which require the students to practise 
manual and intellectual skills and habits, 
and so on. 


The third major task in education is to 
organise the learning experiences to maxi- 
mise their cumulative effect. We can increase 
the cumulative effect of learning experiences 
in two ways: through sequential organisation 
and through integration. When learning experi- 
ences are arranged so that the students begin 
with simpler concepts and skills and go on to 
broader and deeper applications, far greater 
learning results than with a random arrange- 
ment of learning. This is sequential organ- 
isation. Integration mvolves relating what is 
taught m one part of the educational pro- 
gramme to what ts taught in another part. 
Effective organisation of learning experiences 
involves planning both for sequence and for 
integration. 


The fourth major task in education is to 
evaluate the effectiveness of the educational 
programme in attaining its objectives, through 
appraising the educational progress of the 
students. This requires evidence of changes 
in students’ behaviour during the time they are 
taking part in the educational programme. It 
means appraisal early in the course as well as 
near the end. \t involves evidence relating to 
all of the important objectives which will help 


us to identify those aspects of the curriculum 
that are effective and those that need impro- 
ving. 


With this brief review of the essential character- 
istics of a profession and the major tasks in 
planning and conducting an educational 
programme, we can identify more easily the 
distinctive attributes of education for the 
professions. 


Objectives of professional education 


The existence of principles and of a code of 
ethics in a profession points to several kinds 
of educational objectives which are important 
in educating for a profession. The development 
of an ethical practitioner who has an adequate 
understanding of the ethical code of the pro- 
fession, who applies the ethical principles 
intelligently to the varied particular instances 
that arise, and who is sincerely committed to 
the highest ends of the profession, requires an 
education programme which consciously aims 
at several major types of objectives. 


In terms of knowledge and understanding, a 
programme of professional education needs 
to develop in students a broad and clear 
concept of the social role of their profession. 
This includes understanding of the social 
functions the profession serves and how these 
functions are related to the total functioning 
of society and to the functions of other major 
specialised groups. It also includes an under- 
standing of the various kinds of relations 
which exist between the profession and 
society at large and between the profession 
and other specialised groups, including the 
expectations which these groups hold re- 
garding the members of the profession. 


Also,in terms of knowledge and understanding, 
professional education aims at developing a 
deep understanding of the persons to whom 
the professional service is rendered including 
particularly insight into personal motivations, 
feelings, needs, and the interrelation of 
physical, psychological, social, and emotional 
aspects of human behaviour. Furthermore, 
professional education needs to include among 
its objectives the development of self-under- 
standing in the students. Obviously, in no 
sense can complete self-understanding be 
attained by the time of graduation from the 
professional school, but a sufficient beginning 
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can have been achieved to provide for con- 
tinued development as he pursues his pro- 
fessional career. 


In terms of effective thinking or problem- 
solving, the objectives of professional edu- 
cation which are derived from the import- 
ance of ethics include the ability to recognise 
ethical problems, the ability to identify the 
ethical principles at issue, and the ability to 
work out appropriate courses of action in 
terms of ethical principles. 


In terms of attitudes, education for the 
professions aims at developing loyalty to 
the social well-being of the persons who are 
served by the profession, concern for a truly 
social role on the part of the profession, a 
sense of self-respect for the social contri- 
butions of his profession and of his own 
work, and a warm, accepting, yet objective 
attitude towards his clients. This involves 
developing in professional students a con- 
siderable degree of emotional maturity so 
that they are free to express and receive 
emotionally charged communication and at 
the same time can act intelligently as new 
problems arise. 


The foregoing objectives clearly represent a 
large task for professional education, but 
these are not all the important objectives. The 
requirement that the professional practitioner 
operate on the basis of principles rather than 
rules implies certain additional objectives if 
the educational programme is to aim con- 
sciously to develop students who can operate 
in this fashion. Some of the purposes outlined 
above contribute to the performance of pro- 
fessional duties in terms of principles. In 
addition, in terms of knowlege and under- 
standing there is need to understand the 
structure and functioning of the institution 
in which the profession operates, whether this 
be school, church, hospital, court, or other 
institution. Only with such a broad view can 
a professional practitioner Operate with 
intelligence and on the basis of principles 
adequate to produce desired consequences. 


Finally, of course, every professional school 
needs to aim consciously to develop an under- 
standing of those principles, concepts, facts, 
and procedures which are basic to professional 
operations. In medicine these include prin- 
ciples of physiology, anatomy, chemistry, 


3.08 


physics, bacteriology, and psychology. The 
tendency, however, is to limit these basic 
principles too narrowly. In a very real sense, 
doctors deal with problems in their normal 
professional work which are psychological 
and require an understanding of relevant psy- 
chological principles if they are to operate 
intelligently. 


In terms of effective thinking of problem- 
solving, it is clear that the foregoing objectives 
involving knowledge imply the development 
of some skill in recognising professional 
problems, in analysing the problems in terms 
of the relevant principles, and in working out 
courses of action by applying these principles. 


In terms of attitudes, the use of principles, 
rather than rules, in a profession requires as 
objectives in professional education the devel- 
opment of broad, rather than narrow interests 
in the fields on which the profession draws, 
and the development of the student’s interest 
in continuing his own learning long after grad- 
uation from a professional school. 


Planning learning experiences 


The previous section outlines characteristic 
cbjectives of professional education. What 
about the learning experiences which the 
professional may use to attain these objectives? 


The most common problems in professional 
schools in connexion with the learning 
experiences used are: 


= the failure to select learning experiences 
in terms of the objectives to be attained; 


= the failure to utilise consciously appro- 
priate learning procedures for developing 
problem-solving skills, attitudes, and 
interests; and 


= the failure to develop effective motivation 
for learning. 


These are serious deficiencies. 


Since learning is an active process, in which 
the learner himself is definitively involved, 
motivation is essential. Since the learner 
learns more than knowledge of content, and 
be actually learns what be is doin g, what be is 
feeling, what he is thinking, it is important to 
make conscious plans for students to learn to 
solve problems, to develop attitudes and 


interests. Since objectives represent the aims 
of the professional school and _ learning 
experiences are the means of reaching edu- 
cational aims, the latter should be planned in 
terms of these aims. 


An essential factor in planning learning 
experiences in terms of the objectives sought 
is to have clearly in mind what the objectives 
imply, both as regards behaviour to be devel- 
oped and content involved. For example, if the 
objective “understanding the social functions 
the profession serves and how these functions 
are related to the total functioning of society 
and to the functions of other major specialised 
groups” is to serve as a guide for planning 
learning, we must have a clear idea of what is 
meant by “understanding”’ as a type of be- 
haviour to be developed and what content is 
included in the phrase beginning ‘“‘the social 
functions the profession serves’’. Most in- 
structors who have sought to define ‘‘under- 
standing”’ indicate that it is a mental process 
that is more active than memorisation, since 
it involves not only remembering but also the 
ability to explain the concept or principle in 
one’s own words, the ability to mterpret, to 
illustrate, and to compare and contrast it 
to related ideas. Such a definition clarifies 
the behaviour that the student is expected 
to develop and, as pointed out earlier, it 
suggests the kind of learning experiences that 
are required to attain the objective. These 
would be learning experiences which give 
the student opportunity to explam, to inter- 
pret, to illustrate, and to compare and con- 
trast it to related ideas. By defining the 
content implied by the phrase beginning 
“the social functions the profession serves’’, 
we are able to identify what the principles 
and concepts are which the students will be 
given opportunity to explain, to interpret, 
to illustrate and to compare and contrast 
with related ideas. 


Correspondingly, as we define each objective 
in terms of the behaviour and contentimplied, 
it is a much easier step to select learning 
experiences that give students a chance to 
practise the behaviour involved and to utilise 
the relevant content. In this way, learning 
experiences are planned in terms of the 
objectives sought. 


The fairly common practice in professional 
schools of giving almost exclusive attention 


to knowledge and to technical skills may be 
partly due to the failure to recognise that 
effective thinking, attitudes and interests are 
also learned and can be consciously developed 
in programmes of professional education. 
Skills in ways of thinking are acquired by 
practice of solving problems. A new attitude is 
acquired by repeatedly observing and reacting 
to certain phenomena from a new perspective. 
Interests in certain kinds of activities are de- 
veloped as the student gains satisfaction 
from participating in these activities. For each 
of these types of objectives, definite learning 
experience can be provided so that students 
are practising problem-solving, they are pro- 
jected into situations from a new perspective, 
they are able to gain satisfaction from certain 
kinds of activities. In short, professional 
schools can plan learning experiences for 
developing effective thinking, attitudes and 
interests. 


This leads to consideration of the primary 
significance of motivation. Since the learner 
learns through his reactions, unless he can 
be involved in the situation, unless he can be 
guided to think, feel, and act in ways appro- 
priate to the situation, it is not possible for 
him to learn. Practice alone, even when 
carried to unusual limits, does not take the 
place of the learner’s being involved in what 
be is doing. 


Organising learning experiences 


There are not only problems involved in the 
selection of learning experiences but also in 
their organisation. As the programmes of 
professional education have become more 
complex and involved larger staffs, they 
have become more disintegrated. Yet signi- 
ficant changes in behaviour, that is, funda- 
mental learnings, require a long time to 
develop. It is necessary that what is learned 
this term builds upon what was learned last 
term, that what will be learned next year 
builds upon what is learned this year. This is 
sequence learning. 


However, effective sequence is not simply a 
series of repetitions from one year to another. 
Rather, it includes variety m the learning 
experiences, so that each subsequent term 
emphasises the main things to be learned, but 
in varied contexts. In this way, ever broader 
and deeper learnings are achieved. 
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Furthermore, effective organisation provides 
for relating one course to another and one 
field to another, which reinforce the learning 
in each course or field. This is done both by 
helping the student to use things learned in one 
course or field in another, and by helping him 
to perceive differences as well as similarities in 
the concepts, principles, attitudes and skills 
utilised in the various courses and fields. This 
is called curriculum integration. 


Sequence and integration are essential to pro- 
grammes of professional education, but 
because of the tendency towards special- 
isation and separation, conscious efforts are 
required to plan for and develop effective 
organisation. 


One significant development in working out 
a better organised curriculum is the building 
of a closer and more appropriate connexion 
between theory and practice, between the 
art and the science of the profession. Many 
institutions are not only teaching general 
principles but also helping students to apply 
these principles to particular cases so that it 
is possible for them in practice to use the 
principles as they deal with particular cases. 
Effective professional education requires this 
close connexion between theory and practice. 
Without theory, practice becomes chaotic, 
merely a collection of isolated, individual 
cases. Theory gives meaning and unity to 
what would otherwise be specific and isolated 
cases, 


On the other hand, without practice theory 
becomes mere speculation. The realities of 
practice provide a check upon pure 
speculation, a test of the adequacy of theory; 
and practice provides the problems which 
must be dealt with by any comprehensive 
theory. Hence these efforts to connect theory 
and practice more closely are important 
contributions to professional education. 


Another illustration of this development is 
the increasing use of the case method. The 
case method involves the student in the study 
of a concrete and particular case. However, for 
him to understand this case and deal with it 
effectively, he must bring to bear the theory, 
the concepts and the principles that are basic 
to the issues raised by the case. The internship 
provides concrete experiences which are inter- 
preted in terms of basic theory in the accom- 
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panying seminars. This shuttling back and 
forth between general and specific aspects 
of a profession helps to build an increasingly 
adequate context of concepts and principles 
by which a member of the profession can 
understand the operations to be performed 
in relation to the values to be attained. An 
adequate theory helps him to relate particular 
activities in an individual case to the larger 
social issues, to see the connexion between 
the activities of the daily operations of the 
profession and the welfare of society generally. 


However, for these types of educational pro- 
grammes to be effective, more than super- 
ficial experience and explanation must be 
provided. The student needs many oppor- 
tunities to deal with situations on the basis 
of a careful analysis of them, to identify the 
values and principles involved, and through 
practice to develop artistry in devising means 
to deal with the situation in order to preserve 
these values. This is the aim of efforts to knit 
theory and practice more closely together. 


Another effort at extending the sequential 
organisation of professional education is the 
working out of definite plans for continuing 
education after the member of the profession 
has completed pre-service training and bas 
been inducted into the initial activities of 
bis work. Fifty years ago medical schools 
felt that their work had been done when 
their graduates had been admitted to initial 
professional activity. As the years went by it 
became increasingly clear that many members 
of the profession did not grow after they began 
their work and that in some cases the older 
practitioner was much less competent than 
those currently beginning their profession. 
Few doctors made any effort to continue 
their education after graduation. 


Now there is a marked tendency for pro- 
fessional schools to develop programmes of 
continuing education, in some cases bringing 
the work of the school to the practitioner in 
the field, in other cases setting up short 
courses, institutes, or long-term seminars for 
practitioners to take on the campus. 


Evaluating effectiveness of educational pro- 
grammes 


The final attributes of professional education 
are those involved in the task of evaluating 


the effectiveness of the educational programme 
in attaining its objectives, through appraising 
the educational progress of the students. Four 
important attributes are too often neglected 
in current educational programmes. 


The first of these is conducting an appraisal in 
terms of all of the important educational 
objectives of the professional school. The 
common practice is to appraise the knowledge 
of the students and certain of their technical 
skills. In addition, many professional schools 
appraise the student’s ability to solve problems 
as these are presented in verbal form. Few 
institutions provide for careful, systematic 
appraisal of problem-solving in the professional 
situation, and appraisal of professional in- 
terests and attitudes. Hence the school does 
not have a comprehensive picture of the 
achievements of its students in terms of its 
Own purposes. 


A second attribute, often neglected, may 
partly account for the shortcomings in the 
first. A comprehensive programme of eval- 
uation uses varied devices for obtaining evi- 
dence regarding the educational progress of 
students. These devices include not only 
written tests and examinations, useful as they 
are, but also observations, interviews, question- 
naires, reports from the field and samples of 
the student’s work; in short, any device which 
gives valid evidence regarding the significant 
behaviour of the student. Few professional 
schools consistently evaluate their effective- 
ness in such varied ways. They tend to limit 
their appraisal to written examinations and to 
rating forms. This does not provide adequate 
means for comprehensive evaluation. 


Appraisal of the progress of students toward 
the objectives of professional education re- 
quires evaluation at several points in his career. 
This is a third essential attribute. To get 
evidence of progress requires at least three 
appraisals, one early in his attendance at the 
professional school, one near bis graduation, 
and one after several years of service in the 
profession. Some schools attempt annual 
appraisals during the student’s enrolment. 
The changes made while in the school throw 
light on the immediate effectiveness of the 
school’s educational programme, while the 
appraisal after the student has spent several 
years in the profession provides evidence of 


the permanence of learning and the extent to 
which it has achieved some continuity with 
professional experience. Unfortunately these 
studies of progress are quite rare. This may be 
partly due to the lack of appreciation of the 
way in which systematic sampling of students 
and alumni is small enough to permit the use of 
individual interviews yet at the same time 
sufficiently representative to permit valid 
generalisations regarding the populations from 
which the samples were drawn. 


The last attribute to be mentioned here is the 
use of evaluation in improving the educational 
programme as well as in providing information 
to guide work with individual students. Too 
often, what little appraisal of student achieve- 
ment is conducted results only in grades for 
the students. Actually an evaluation pro- 
gramme can serve as a helpful means for 
continued improvement and development of 
the professional school. The results of appraisal 
indicate the respects in which students are 
making substantial progress and the respects 
in which expected development is not taking 
place. These suggest aspects of the educational 
programme which need re-thinking and re- 
planning to provide for improvement. Further- 
more, as revisions are made in the programme, 
subsequent evaluation indicates the relative 
effectiveness of these revisions. Hence, 
appraisal provides a sound basis for planning. 


The evaluation data also indicate the progress 
made by individual students and bring to 
attention both their strength and their diffic- 
ulties. This information thus provides a sound 
basis for the guidance of individuals and gives 
a more substantial foundation for the indivi- 
dual student’s continued planning of his own 
education. 


In summary, the distinctive characteristics of 
a profession, namely its ethical code and its 
operating basis on principles, suggest the 
distinctive attributes of education for the 
profession. From these characteristics, im- 
portant educational objectives can be derived. 
Because these objectives are complex and 
involve understanding, problem-solving, atti- 
tudes and skills, they require clear definition 
in order to develop effective methods for 
their attainment. The difficulty of attaining 
the goals of professional education makes 
motivation of prime importance and effect- 
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ive organisation of learning experience a for constructive work with individual students. 


necessity. Finally, a comprehensive evaluation The building of an effective programme of 
programme is required to guide the continued education for a profession 1s not easy but 
development and improvement of the educa- when attacked intelligently, systematically, 
tional programme and to provide information and enthusiastically, it can be done. 
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The author of the preceding pages* is Ralph W. Tyler. 
He obtained his doctor’s degree from the University of 
Chicago in 1927 ... more than half a century ago! 
He wrote these lines in 1951... more than a quarter 
of a century ago! yet his vision is still valid today and 
still remains to be acted upon. 
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For those concerned in curriculum planning this clearly 
means that the order of the day is patience and 
perseverance. 
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*The paper was presented on 30 January, 1952, in 
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of Social Work and printed in the Social Work Journal April pars > Saini ee Association of Schools 
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programme describes 


=# a series of planned educational activities a 
student is to go through with the assistance 
of teachers 


integration 
(of a curriculum) 


= coordination of different teaching/learning 
activities to ensure the harmonious functioning 
of the educational process for more effective 
training 


the four c’s of curriculum planning 
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cooperative 


continuous 


comprehensive 


concrete 


A programme prepared jointly 
by a group of persons will be 
less liable to error than one 
prepared by a single person. 


The preparation of a pro- 
gramme is not a one-shot 
operation. In planning it, 
provision should be made for 
its continuing revision. 


In an approach which accepts 
the interaction of all the pro- 
gramme components must be 
defined with the requisite pre- 
cision. 


General and abstract consider- 
ations are not a sufficient basis 
for drawing up a programme. 
Concrete professional tasks 
must constitute the essential 
structure of a__ relevant 


programme. 
from E. Krug. 


plan of action for preparation 
of an educational programme 


Executing body Advisory body 


Ministries of University staff, 


Chronological 


order Definition of objective 


al 


Prospective study to evaluate the 


country’s requirements Planning, national and inter- 
(quantitative and qualitative) Health, national experts 
in respect of health personnel, Education, 


taking into account what is etc. 


available and can be absorbed 


Ministries of 
Education 
and Health 


Definition of the educational 
objectives of the school of 

health sciences in accordance 

with the tasks to be accomplished 
by each professional health worker 


Special faculty 
commission 


Organisation of an information Faculty staff Faculty of 
group on education planning for 7 Pedagogy Education 
health sciences consultant 


Acceptance by the faculty staff of the goals established in the light of the 
country’s requirements; training of teachers in education planning 


Evaluation 
committee 


Department of 
Education 
Education consultant 


Development of a system of 
evaluation: 


m certifying tests 


® continuous formative evaluation 
indicating whether the goal has 
been attained 


Faculty of Pedagogy 
Education consultant 


Programme 
committee 


Modification of programme 
and selection of methods by 
which to attain the goals 


Organisation of a group for research | Faculty staff Faculty of Pedagogy 
in education 


Periodic re-evaluation of goals and __| Ministries of University staff, 
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methods in response to changes in Planning, national and inter- 
the country’s needs and in thelight | Health, national experts 
of evaluation Education, 
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Diagram showing factors influencing curriculum design 


Health Needs 
of Society 


Health 
Services 


Educational Tasks Revision 
and Objectives of Objectives 


Selection 


Teaching and 


Learning Activities Graduates 


Students 


Research in Testing 
and Measurement 


Students Environment Faculty Organization 


Budget, Health Situation, etc. Budget, Educational 
Competence of Teachers 


Action of Manipulatable Variables 


Working Hypothesis Revision of Hypotheses 
Concerning Curriculum Concerning Curriculum 
Preparation Preparation 


the purpose of teaching 
is to facilitate learning 


The literature on the philosophy of education 
is rich in theories, which tell a story of timid 
steps forward, backward leaps and rediscover- 
ies. It would be very gratifying to have a 
reliable general theory, firmly seated on a 
scientific basis and making proper allowance 
for social variables, which could serve as a 
guide for every teacher and enable him to 
resolve the ‘“‘real’’ problems of teaching the 
health professions. Unfortunately, such a 
universally satisfactory general theory does 
not exist. On the other hand, by means of 
the systematic approach suggested, bhypo- 
theses can be formulated regarding the 
process of acquiring a satisfactory level of 
performance that can be evaluated empiri- 
cally and the choice of learning activities 
facilitated. 


The contemporary trend is to stress the 
“teaching-learning system” as opposed to the 
preponderance previously given to teaching 
alone. There is a tendency to be interested 
less in teaching than in learning, less in what 
the teacher presents and more in what the 
student learns. Lawrence M. Stolurow has 
criticised what he calls the ‘““ccommunication- 
learning fallacy” which assumes that the 
information transmitted to the student is 
always learned. Doubtless this is obviously 
fallacious, and known to be so when it is 
expressed so crudely, but discussions of 
teaching methods are often still inspired by 
it and it has even been carried over into the 
initial stages of research into new methods. 
Much of this research, by concentrating on 
problems of the presentation of stimulating 
materials and utilising some of the more 
rudimentary concepts of communications 
theory, dealt too much with the manner in 
which information was transmitted to the 
student without investigating very closely 
what was learned and by whom, at what speed 
and, i particular, for what purpose. Under- 
standably, this approach led to a passive 
attitude towards students’ response; the 
student was seen in a dependent situation, 
relying upon information directed at him, 
whether through modern audiovisual com- 


; Adapted from Norman Mackenzie et al. 
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munication techniques or the more traditional 
forms of the lecture and the text book. 


Learning, however, is a dynamic and inter- 
active process in which the behaviour and 
experience of the student are vital components; 
the student must not only receive but also 
contribute ; his perception of what is happening 
is just as important as the perception of his 
teachers and the assessment he makes of the 
value of a learning activity may be more 
relevant than that of his examiners. Good 
conventional teaching, of course, has always 
sought to take account of the learner, but its 
structure and methods have greatly inhibited 
it. The rigid style imposed by large numbers, 
timetable requirements and the availability of 
teaching space, by the conventional practices 
in designing courses and by teaching confor- 
ming to an accepted academic discipline, have 
led to the “teaching” aspect again dominating 
over the “‘learning’’ aspect. If it is accepted 
that the starting point must be the acquisition 
of knowledge rather than its communication 
then we must ask some different questions 
and intensify our investigation of non-directive 
methods. Using the same approach as 
Jerome S. Bruner we can consider what ex- 
periences will motivate the student and enable 
him to learn, in what ways knowledge can best 
be structured for a given student or group of 
students, what sequence and in what form the 
material can be presented most effectively, 
what should be the nature and the frequency 
of rewards or penalties, and how we can grad- 
ually lead a student to give less thought to 
extrinsic rewards than to the personal satis- 


faction of having achieved a desired degree of 
skill. 


The body of knowledge possessed by a group 
of teachers or set out in a series of authorit- 
ative volumes is the fruit of intense intellect- 
ual activity. Teaching a so-called basic science 
is not a matter of getting the student to 
memorise it, but rather of helping him to 
participate in a process that renders the ac- 
quisition of a body of knowledge possible. A 
subject is taught not to produce little living 


Teaching and learning, Paris, UNESCO, 1970, pp. 44—50. 


libraries on that subject but rather to get the 
student to think for himself in accordance 
with the laws of physics, to consider pro- 
blems from the same angle as the biologist and 
to assimilate the process of acquiring know- 
ledge. Knowing is a process, not a product. 


If the revision and renewal of the programme 
is regarded as part of the teaching-learning 
process, a change must also occur in the roles 
and interrelationships of teachers, students 
and others who are concerned. The system 
becomes much more complex than the con- 
ventional linear model and it demands much 
more from those who have to design, manage 
and operate it. To put it epigrammatically, 
the teacher becomes a learner himself, and the 


learner undertakes some part of the teaching 
role. This is because the teacher learns more 
about teaching and the student begins to 
assume a greater responsibility for his own 
progress. The roles of other participants also 
change and new roles must be added to the 
cld. Some of the new methods at present under 
trial, for instance, not only require librarians 
to emerge from their traditional sphere into a 
more dynamic type of activity; they also call 
for the services of highly qualified personnel, 
such as artists, photographers and technicians, 
as well as educational technologists and psy- 
chologists. In fact, the ecology of a teaching 
institution changes once its primary function 
1s redefined, namely to facilitate the acquire- 
ment of competence by the student. 
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In the following pages a certain number of definitions, 
epigrams and lists are proposed whose aim is to serve, 
where appropriate, as a starting point for reflections or 
discussions on the methodology of education. These 
elementary concepts are derived from various systems, 
and none of them is regarded as embodying an absolute 
and final truth. 
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Always assume that a teaching activity is ineffective 
unless there is evidence to the contrary. 


Adapted from A.L. Cochrane 
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teaching 


Interactions between teacher and 
student under the teacher's res- 
ponsibility in order to bring about 
expected changes in the student's 


behaviour. 


purpose of teaching 


to help students to 


=“ acquire, retain and be able to use knowledge 

= understand, analyse, synthesize and evaluate 
= ~=achieve skills 

# establish habits 


= develop attitudes 


teaching approaches 


# @# talk to students 

= talk with students 

a have them talk together 
= show students how 

= supervise them 


= provide opportunities for practice 


S. Abrahamson 
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Teaching methods which place the student in an active 
situation for learning are more likely to be effective 
than those which do not. 

G.E. Miller 
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The activities composing a training programme must 
be organized so that students can acquire their pro- 
fessional skills under conditions similar to those in 


which they will later practise. 
Mememee ees: Le) sole 2B} By Ge Be Beh eisteSeas 
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If only ideal or model learning situations are used, the 
student will be deprived of the stimulus of having to 
face realities and will have less opportunity to propose 
improvements with the help of his fellow-students and 
his teachers. 
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Learning is both an emotional and an intellectual process. 
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learning 


A process” resulting in some 
modification, relatively permanent, 
of the way of 


thinking, 
feeling, 

doing, 

of the learner. 


the characteristics of learning 


learning is 


# producing a behavioural change in the learner 


= leading to a relatively permanent change that 
is also gradual, adaptable and selective 


= resulting from _ practice, repetitions and 
experience 


= not directly observable 


some principles of learning 


= learning is individual 
# motivation is the key 


= relevance of learning experience should be 
clear to the student* 


mw “feedback” to learner is important 


* see paragraph 2 page 2.14 


iad 


conditions to facilitate learning 


An atmosphere which 


encourages people to be active 

emphasizes the personal nature of learning 
accepts that difference is desirable 
recognizes people’s right to make mistakes 
tolerates imperfection 


encourages openness of mind and trust in 
self 


makes people feel respected and accepted 
facilitates discovery 
puts emphasis on self evaluation in cooperation 


permits confrontation of ideas 


G.P. Pine & PJ. Horne, Principles and Conditions for Learning in 
Adult Education — Adult Leadership. Oct. 69 
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learning 


# is primarily controlled by the learner 

is unique and individual 

= is affected by the total state of the learner 
zB is cooperative and collaborative 

i is an evolutionary process 

@ is a consequence of experience 


% is not directly observable 


G.J.Pine & P.J. Horne (ibid.) 
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1. Complete in less than five lines the following sentence: 


“1 learn best when | 


Ask several of your colleagues to do the 


same exercise; make a list of all the 
situations. 


In the educational institution to which | 
belong, the following learning situations are 
_ provided for students: 


Note the diversity of situations and 
compare it with those provided for 
students by the educational institution 
to which you belong. 


4. \f the comparison shows that changes 
are needed in your institution, read the 


following pages which describe the 
teacher’s functions, and do the exercise 
on page 3.39. 
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teaching: a complex but challenging 


task 


The teaching of health personnel has become more complex than it used to be, especially 
if it is accepted that the teacher should consider the benefit of the learner and not only 
his own satisfaction. More than a dozen of the teacher’s functions are described below. 
A teacher should feel satisfied if he fulfils only a few of them, because the present educa- 
tional environment does not greatly help him along this road. Nevertheless, he will do 
well to improve his performance by training as an educator. 


It is sometimes thought that a teacher’s 
activities are so intricately related to the local 
culture in which he functions that the de- 
scription of his tasks should differ if he 
teaches in Antwerp, DaresSalaam_ or 
Leningrad. However there is no support for 
this concept in reality. Data concerning 
teachers’ behaviour during teacher training 
workshops organised by the World Health 
Organization over the last 10 years with parti- 
cipants from a great variety of geographical or 
socio-political backgrounds clearly indicate 
that teachers are more like each other than 
they are like the people of the community 
to which they belong. The worldwide academic 
community seems to be one cohesive entity 
modelled on nineteenth century Western 
European concepts of academic freedom, 
exclusiveness and detachment. 


Most teachers in schools for health personnel 
conduct scientific research, write reports and 
articles, plan and supervise laboratory or 
clinical units and, finally, teach, in that order 
of priority. To teach means, in most instances, 
to prepare and deliver lectures, to supervise 
students during laboratory and clinical work, 
to decide which questions should be asked at 
examinations, and to score papers and oral 
examinations. Added to all this is attendance 
at numerous scientific or administrative 
meetings. 


Instructional duties are sometimes called 
“teaching load”, an expression which clearly 
conveys the attitude to that function. Teaching 
staff are jokingly depicted as secretly be- 
lieving that academic life would be much 
easier if there were no students. As in all 
jokes, there is much truth in this. 


Some teachers are seldom seen by students 
while others spend many pleasant hours in 
stimulating discussion with their students. 


Rather than pursue this description, in which 
no teacher would ever want to recognise him- 
self, let us consider what are the teacher’s 
functions in the light of present-day educa- 
tional concepts. 


1. To be available 


The teacher should be available to give students 
constructive criticism of their working ob- 
jectives and methods. The key-word here is 
“available’’. It contrasts sharply with the habit 
of the traditional academician who proclaims 
“I am always happy to meet students . . . my 
door is always open”, but who, in fact, is so 
rarely in his office that only very stubborn 
and highly motivated students manage to 
catch him once he has left the lecture hall. 
Really available is the professor who has a 
notice on his office door saying “I am at the 
disposal of students in this office every 
Tuesday and Friday from 10 to 12 a.m.”, and 
who is actually there at those times. 


2. To provide constructive criticism of the 
student’s learning objectives and 
working methods 


It isof the utmost importance that the student’s 
working objectives and methods should be 
exposed to constructive criticism as, after 
training, health personnel should be able to 
relate their own professional goals to the 
organised system of health care, and to 
develop their own tactics for achieving those 
goals. 


? 
It is also important to develop the student S 
desire to seek criticism, supervision and advice 
from teachers and fellow-students. 


3. To analyse and evaluate health problems 


Generally speaking, clinical teachers are able 
to guide students in solving patients’ particular 
problems, but students should also collect 
and use relevant information for the solution 
of community health problems. It is less 
certain that all teachers have enough under- 
standing of epidemiology and of health man- 
power planning principles and methods to 
assist students in that respect. This is especially 
true of teachers of the so-called basic science 
subjects. 


It would be an advantage if all teachers had 
sufficient competence to ensure that what- 
ever they teach is relevant to the community 
health situation. The teacher should be able 
to analyse and evaluate health problems 
raised by students, as well as proposed so- 
lutions. 


4. To define learning objectives 


The teacher should be able to define the 
abilities that students need to acquire in order 
to solve a given health problem. For example 
a teacher should be able to define explicit 
and relevant learning objectives so as to help 
students to understand a problem. Moreover, 
when programmes are being constructed, it 
is preferable to indicate clearly the standard 
of performance to be acquired by students 
than to state only the amount of time to be 
spent teaching the subject. It is well known 
that the time needed to master a subject 
varies from student to student depending on 
his ability, motivation and drive, and on the 
availability of teachers and educational facili- 
ties. What is important for the average person 
is not the length of time during which teachers 
teach or students learn, but the ability of the 
person providing health care to perform a 
given task. If my son needs a hernia operation, 
I do not wish to be told that the surgeon has 
graduated from a six-year course in an 
institution in the WHO World Directory of 
Medical Schools. What I want to know is 
whether he is competent to perform that 
operation in the best and safest way. As for 
the nurse who will comfort him in this strange 
and frightening hospital world, I do not care 
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whether she has had 72 hours in a psychology 
course as long as she is able to give him the 
reassurance he may need. 


5. To assess students’ work 


The teacher should be able to define criteria 
for assessing students’ work and measuring 
their abilities. Such criteria need to be care- 
fully drawn up, and teachers should share 
the task of preparing them with their students. 
The sad state of most examination systems in 
faculties and schools is largely the result of 
the absence of such criteria, which could 
serve not only for certification purposes but 
also in guiding the students’ learning, a point 
that is certainly no less important. Everybody 
in the health professions should cultivate this 
ability to define criteria, as it is essential that 
they should be able to audit their own perfor- 
mance and that of their colleagues. 


6. To prepare learning aids 


The teacher should prepare learning aids and 
materials, provide information that will 
facilitate students’ independent work, and 
evaluate the usefulness of such materials. 
Most of what is usually transmitted by 
traditional lecture courses could be better 
acquired by students working independently 
and studying books, scientific journals and 
duplicated texts in self-learning format, with 
or without audiovisual aids. In such cases, 
students should be aware of related learning 
objectives, and feedback evaluation exercises, 
based on those objectives, should be available. 


The preparation of such materials may well 
be a joint effort among colleagues from several 
faculties. It calls for a lot of work, but may be 
considered as an investment that will pay off 
during several years through setting up banks. 
It should help to eliminate passive learning 
which is often all that is required of students. 
The assessment of the usefulness of such 
materials, which requires a level of educational 
competence some teachers do not yet have, 


may call for the collaboration of specialists in 
education. 


7. To select professional activities for 
students 


The teacher should select professional activities 
of graded difficulty appropriate to the 


student’s level of progress during the whole of 
his learning curriculum. There is agreement 
among professional educators that students 
should acquire professional competence 
through the practice of real (or, if that is not 
possible, simulated) professional activities. 
Even students fresh from secondary school 
are perfectly capable of drawing valuable 
lessons from the study of real cases. 


8. To confront students with new problems 


The teacher should confront students with a 
diversity of new problems in order to help 
them define the terms of the problems and 
find solutions to them. If it is true that the 
estimated half-life of medical knowledge is 
about five years, and is slowly but constantly 
shrinking, it is absolutely essential that today’s 
graduates, among whom are tomorrow’s 
leaders of health services, should be able to 
solve the new problems that arise m new 
situations. It is here that the basic sciences 
have an important role to play as they can 
help the student to acquire a scientific method 
of inquiry. Such an approach is essential to 
keep knowledge and skills up to date. The 
fact that the graduate will have to replace 
much of what he knows is the least part of 
the problem; the main difficulty is that we, 
the teachers, do not know which half. 


9. To develop problem solving skills 


The teacher should facilitate students’ learning 
by asking questions requiring problem solving 
skills rather than simple recall of facts. Even 
the most traditional teacher will agree that 
students must acquire basic skills, either 
manual (e.g. inserting an intravenous drip) or 
intellectual (e.g. taking a history), should be- 
have in an appropriate manner with the patient 
and his family, and should not merely possess 
theoretical knowledge. 


In spite of this, whenever the final examin- 
ations used in faculties and schools are ana- 
lysed, it is found that a majority of the 
questions test only simple recall and not the 
intellectual skill of interpreting data and 
solving problems. No attempt is made to 
measure practical skills and attitudes. The 
fact that most examinations require students 
to recall facts without recourse to books is 
highly significant. Of course, health personnel 
must possess a large amount of knowledge that 


they can call upon without reference to books, 
otherwise their usefulness in an emergency 
situation would be seriously impaired. 


Teachers should therefore test students on 
their ability to recall those facts that will be 
needed immediately in emergencies and have 
been clearly specified as such from the start. 
Apart from this, examinations should con- 
front students with typical individual or 
community health problems and allow them 
access to books, in accordance with the sound 
practice adopted in everyday professional life. 


10. To aid the understanding of basic scienti- 
fic principles 


The teacher should endeavour to ensure that 
students understand the basic principles under- 
lying the activities and tasks they are learning 
to master. Factual details can be readily added 
to an understanding of basic principles and 
concepts, but such principles are not easily 
derived from a mass of facts. Students usually 
learn the so-called basic sciences early in the 
curriculum before they have met the real-life 
problems which require an understanding of 
those subjects. They are thus poorly motivated 
and soon forget what they have learned. It has 
repeatedly been found that students tested 18 
to 24 months after having followed a basic 
sciences course answered at random, just as 
if they had never studied the subject. 


11. To supervise the student’s progress 


The teacher should check each step the stu- 
dent takes in solving health problems in order 
to discover the areas where he fails to take 
account of the relevant basic scientific prin- 
ciples. This role follows naturally from the 
previous one. As mentioned above, in the 
traditional curriculum students are taught 
basic sciences in the first two or three years. 
Ideally, a student should be helped to reach 
an acceptable solution to problems through 
his understanding of the principles of the 
basic sciences and by adopting a scientific 
approach. One of the persons best qualified to 
judge whether the student is thinking in a 
scientific manner is the basic sciences teacher. 
In the training of a nurse or of a general 
practitioner, for example, the role of the 
basic sciences teacher is not to instruct in a 
given subject but to ensure that the nurse and 
the physician are capable of applying basic 


3.35 


scientific principles in their daily work. 


A physiologist could spend two hours a week 
in a ward or an outpatient clinic with his 
students, going through the medical records 
of current patients to check whether basic 
principles had been adhered to, and draw 
the students’ attention to unsatisfactory 
examples. Such an association between basic 
sciences and clinical activities would be of 
greater value than the present so-called inte- 
grated forms of teaching in which teams or 
panels of teachers endeavour to teach in a 
coordinated manner. 


12. To identify the factors underlying health 
problems 


The teacher should make sure that the student 
identifies and takes account of the psycholo- 
gical, cultural and socioeconomic factors 
underlying the health problems with which 
he deals. 


In practice, there is no natural frontier between 
the physical sciences and the humanities, nor 
between science and art in the broadest sense 
of the words. Giving courses in the humanities 
and the behavioural sciences does not auto- 
matically ensure that future health workers 
will be able to relate their daily activity to 
the culture to which they belong. They are 
more likely to do so if they are required in 
practice to define the patient’s social and 
psychological problems in relation to his 
health problems, and to seek solutions, with 
the help of suitable specialists. 


13. To encourage intellectual discipline 


The teacher should encourage intellectual 
discipline by observing students’ work and 
evaluating their activities in relation to 
defined standards of quality of care. 


Both teachers and students pass much time, 
more or less usefully, in large lecture halls. A 
better distribution and use of teachers’ time, 
and the practice of peer-teaching to enliven 
the course, could improve both the quantity 
and the quality of the instruction given and 
allow for more individualised supervision. 


The quality and satisfactory performance of 
learning should be the only yardstick or 
criterion used by the teacher, to the exclusion 
of the time constant (“the duration of the 
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curriculum is six academic years”). What 
should vary is the number of different types 
of task that any given kind of health pro- 
fessional can perform. Every member of the 
health team, and the physician most of all, 
is a potent therapeutic agent but a poorly 
tested one. The least toxic medication listed 
in the International Pharmacopoeia has under- 
gone more thorough double-blind testing than 
any health professional anywhere, at any time. 
Cigarettes are sold with a printed health warn- 
ing. Does not the university have a social duty 
to protect all citizens from the potential 
danger that physicians and the other members 
of the health team may represent? 


14. To set an example 


Finally, the teacher should serve as a model of 
professional conscientiousness, reliability , ana- 
lytical sense and efficiency. We have all met 
such men in the course of our studies. We have 
also met the other kind. We tried to imitate 
the former and sometimes despised the latter. 
In connexion with students’ values and person- 
ality, the following quotation from the 
American Association of Medical Colleges Lon- 
gitudinal Study, reported by E. B. Hutchins 
(1964), should be remembered. Describing the 
effects on students of their years of study, 
Hutchins says “. . . the interest in understan- 
ding how others feel about problems or the 
need to place one’s self in another’s shoes .. . . 
has, on the average, decreased’’. To a potential 
patient, this statement is very disturbing. If it 
reflects a widespread reality, the cause could 
lie in the example that has been set. 


Conclusions 


Nostalgic remarks are often heard about the 
good old days before professional educators 
had begun to invade the educational black 
box. One could then lecture and enjoy the 
feeling of satisfaction any good teacher has 
the right to expect. Nowadays, teaching is 
somewhat more complex, especially if the 
idea is accepted that it should be for the 
benefit of the learner and not only for the 
satisfaction of the teacher. Some teachers 
have already undertaken some of the functions 
described in this paper. They may feel satisfied 


even if they fulfil only a few, as the present 
educational environment is of no great help 


to their efforts in that direction. The con- 
clusion must be that, despite all that has been 


achieved in the field of teacher training, much ing as educators, even though such courses are 
still remains to be done. Some have more gift still too seldom sought after and too seldom 
for teaching than others, but all have the possi- available. 

bility of improving their performance by train- 
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How many teachers are really prepared to have students 
reach conclusions different from their own, giving first 
consideration to the logic of reasoning rather than the 
ability to arrive at a predetermined solution? 


The student body represents the largest untapped 
educational resource in most schools. 


G. Miller (1977) 
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Read, or reread, carefully the description of the teacher’s function, (pp. 3.33 — 3.37) and circle 
the numbers of those that you believe yourself able to perform in your present situation taking 
into account the university regulations and any other local constraints. For each function, describe 
the precise activities that you personally will be able to start or continue during the 12 months to 
come. 


Description of precise activities for each function 
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[exercise (Continued from p. 3.39) 


Function Oe: : Mass 
No Description of precise activities for each function 
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teaching techniques 


The Handbook does not give detailed descrip- 
tions of the different teaching techniques 
employed in centres for training health 
personnel. This is intentional, for an abun- 
dant literature is available on these subjects to 
which the reader is referred (see references), 
Another reason is that the main purpose has 
been to stress the relevant aim of the teaching 
(whence the almost obsessional insistence on 
the need to define educational objectives) 
(chapter 1) and on the means for checking if 
and how these aims have been achieved 
(chapters 2 and 4). The problem is not merely 
one of communicating better (this is the sub- 
ject of chapter 3) but of better communicating 
a relevant message. 


For details concerning teaching techniques, 
lectures, seminars, tutorials, practicals, pro- 
grammed learning, etc., the reader should 
refer to the specialised publications (see 


references, page 7.01). The following are also 
suggested as useful reference documents: 


The lecture method of instruction — Eileen 
Bughman, Public Health Papers No. 52, WHO, 
pp. 57—63 


Audiovisual aids to learning — E. B. Penta 
and T.V.Telder, Public Health Papers No. 
52, WHO, pp. 40—51 


The selection of teaching/learning materials 
in health sciences education. WHO Technical 
Report Series No. 538 


Audiovisual media in medical teaching — 
M. A.C. Dowling, Public Health Papers No. 47, 
WHO, pp. 53—78 


Group techniques in education, UNESCO, 
Educational Studies and Documents Series, 
No. 24, Paris 1977 
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As ‘a teaching technique workshops are certainly very 
useful and efficient. For details see chapter 5. 
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The teacher’s role is to encourage the desire to learn 
rather than the desire to know. 
noooooooooooooooooeoeoeooooood 
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advantages and disadvantages of 
certain teaching methods and of 


different educational media 


1. Apparent saving of time (for the 1. Keeps the student in a passive situation. 


teacher) and resources. 


2. Does not facilitate learning how to solve 
problems. 


3. Offers hardly any possibility of checking 
learning progress. 


2. Presence of the teacher (showmanship). 


3. Covers a large group of students. 


4. Gives a feeling of security. ak 
, : ‘ 4. Does not allow for individual pace of 


learning. 


5. Low receptivity. 


2. Small group activities 


1. Permits a teacher/student dialogue 1. High costs in personnel and time (unless 
(thanks to the availability of the teacher). peer-teaching is used). 


2. Facilitates evaluation. 


3. Demonstrations 


1. The teacher is present (showmanship). 1. Number of students is limited. 

2. Grves a feeling of security. 2. Keeps the students in a passive situation 
3. Develops the quality of observation. (except for observation). 

4. Ensures closer contact with concrete 3. Offers little possibility of checking the 
problems. learning process. 

5. Facilitates the acquisition of practical, 4. Does not allow for individual paces of 
intellectual and communication skills. learning. 

6. Presents reality, not substitutes. 5. High cost in personnel and time. 

7. Enables logical step by step presentation. 6. Difficulty in repeating the demonstration 


8. Is attention catching. in order to acquire competence. 


9. Demonstrates the right way of doing a 
(complex) task. 


10. Presented facing the audience, 
11. Makes it possible to ask questions. 


12. Limits damage to equipment and material 
when students do practical work afterwards. 
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Advantages Disadvantages 


4. Practical work 5. Bedside teaching 6. Field work 


1. Puts the student in an active situation. 1. High personnel, transport and material 
costs. 


2. Permits evaluation of degree to which 
educational objectives (practical and com- 2. Covers a limited group of students. 
munication skills) have been attained. 3 


Sometimes puts the patient in a diffic- 
3. Develops qualities of observation and ult situation. 


na re ae fats 
decision-taking 4. Poor standardisation. 


4. Ensures closer contact with reality 5 
(professional, health situation of country, 
colleagues and teachers). 


Narrow limits of utilisation, therefore 
requiring careful planning. 


5. Permits comparison between practice 
and theory. 


6. Enables student to develop self-confi- 
dence. 


7. Increases diversity. 


7. Role playing 


1. In such a way students have an oppor- 1. This technique is time consuming and 
tunity, in a controlled situation, to develop is strongly dependent on student’s imagina- 
real communication skills and obtain con- tion. 

structive feedback from peers and teachers. 


2. This technique helps students through 
emotion laden situations. It provides them 
with opportunity to be in somebody else’s 
role and develop empathy and understand- 


ing. 
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Books, handouts, programmed learning from books, and simulation (self learning 
packages). 


Enables student to work at his own pace. 1. Necessitates special educational com- 


Facilitates self-evaluation. petence. 


2. High additional investment costs (in 


3. Makes teachi ible with high 
akes mass teaching possible with hig teachers’ time and money). 


efficiency. 


4. High availability. 3. No group dynamics. 


5. Facilitates decision-taking (solution of 
complex problems). 


6. Reduces risks (for patient or society). 


7. Avoids bias transmitted by ‘‘bad tea- 
chers”’. 


8. Allows a good teacher to save time that 
can then be spent on more complex 
activities such as interpersonal relations. 


9. Can be kept up to date with new scien- 
tific developments and contain references 
to other documents. 


9. Real objects and specimens 


1. Present reality, not substitutes. 1. May not be easily obtainable. 


2. Three dimensional. 2. Inconvenience of size — danger in use. 
3. Permit use of all senses in study. 3. Costly or not expendable. 
Usually only usable in small groups. 
Sometimes easily damaged. 


Problems of storage. 


10. Models and simulation devices 


1. Craftsmanship required for local con- 


1. Three dimensional and concept of real- 
struction. 


ity. 


2. Size allows close examination. Simulation models often expensive. 


3. Good for magnified situation. . Usable for small groups. 

(e.g. middle ear mechanism). . Models often easily damaged. 

4. Can be used to demonstrate function as _ Never same as performing technique on 
well as construction. a patient. Beware of faulty learning. 


5. Can permit learning and practice of 
different technique. 


6. Some can be made with local material. 
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11. Graphics (charts, diagrams, schematic drawings), posters, paintings, photographic prints 


1. For small audiences only (unless pro- 
jected with epidiascope). 


1. Promote correlation of information. 


2. Assist organisation of material. 


2. For effective use, good duplicating 


h 
3. Photographs nearer to reality than equipment and trained staff needed. 


drawings, but association often valuable. 


4. Usually easily produced and duplicated 
(black and white photos). 


5. Easy to store, catalogue and retrieve. 


12. Blackboard or flipchart 


1. Inexpensive, can be made locally. 1. Back to audience. 


2. Usable for wide range of graphic re- 2. Audience limited to 50 or so. 


presentation. 


3. Careful drawings erased, not preserved 
for future use, except in the case of flip- 
charts. 


3. Allows step-by-step build up, or organ- 
isation of structure or concept. 


4. Considerable skill required for effective 
use (rarely taught to teachers). 


13. Flannelboard (flannelgraph). (Most of the comments apply also to magnetic board). 


1. May be used repeatedly. 


1. For limited audience only. 


2. Usually preparable from locally available 


2. Difficult technique to use convincingly. 
materials. 


3. Good for showing changing relation- 
ships. 


4. Holds attention if well used. 


5. Can be adapted for group participation. 


Projectable Media 
14. Still pictures -Opaque projection (epidiascope) 


a Enlargement of drawn or printed 1. Demands total darkness for clear pro- 
materials for large audiences. jection (except with very expensive models). 
2. Obviate need for producing slides and 2. Bulky machine, difficult to transport. 


transparencies. 


3. Electricity required. 


3. Enlarged image may be transferred to 
chart or blackboard for copying. 


4. Small objects and specimens may be pro- 
jected. 
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Advantages Disadvantages 


15. Transparencies for overhead projection 


1. Projectable in full daylight to large 
audiences. 


1. Electricity required. 


2. Equipment and materials for making 
sophisticated transparencies expensive. 


2. Presented facing audience. 


3. Relatively easy to prepare with local 


3. Not usually suitable for photographic 
materials. 


material due to cost (although adaptor avail- 
able to take 35 mm slides). 


4. Subjects can be drawn in advance or 
developed by stages with the group. 4. Usually restricted to teacher use, as it 


5. Can demonstrate movements, processes, is not easy to adapt for the learner to use. 


etc. with transparent or coloured perspex 
models. 


16. Slides and filmstrips 


1. Suitable for large audiences. Fixed order of frames in filmstrip res- 


trictive in use. . 


2. Relatively easy production and (in black 
and white) reproduction. 


2. Need partial darkness for viewing unless 


3. Cheapest current forms of visual med- Sip ebsis SES MENICETS lt 
ium. | 3. Duplication of colour slides expensive 


(even impossible in many countries). 


4. Easily adaptable to self-learning packages. 


5. Equipment available for viewing or pro- 
jection without electricity source (battery 
only). 


17. Microfiches 


1. Too small for clear naked-eye viewing. 


1. Easy storage and cataloguing of large 
numbers of visuals. 


2. Although inexpensive equipment avail- 
able for individual use, large group pro- 
jection equipment not readily available. 


2. Exchange of information on available 
collections. 


3. Very cheap per image if projection can 
be assured for large groups. 


4. Small and light for easy despatch. 


18. Films (8 mm and 16 mm) 


1. Close to reality with movement and Does not permit self-pacing. 


sound. 


Films costly and difficult to produce. 


1 
Ai 
2. Suitable for large audiences (16 mm); | 3 dividual films relatively expensive. 
for small groups only (8 mm). 
: ' 4. Electricity required. 
3. Compression of time and space. 

: 5. Equipment difficult to transport. 
4. Emotive, can develop attitudes, pose 


problems, demonstrate skills. 


i.e cr Oh | 
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18. Films (8 mm and 16 mm) (continued) 


6. Darkness needed for viewing (except 


5. 8mm _ loops useful for individual 
rear screen use). 


instruction. 
6. Good learning source if preceded by 
teacher’s introduction and followed by 
discussion. 


7. Imported film may contain inappro- 
priate information (see item in advantage 
6). 


19. Open circuit television 


1. Programme expensive to produce and 


1. Adaptable to large and small audiences 
demands highly skilled staff. 


in widely distributed area. 


2. Receiving equipment expensive and 
difficult to maintain. 


2. Capable of gaining and maintaining 
attention. 


3. Can stimulate emotions, build attitudes 3. Electricity required. 


and develop problems. 


4. No immediate interaction or feedback. 


4. Can conserve resources of instructors 
by simultaneous broadcast to many classes. 


5. Learner must adapt to a set programme 
with no possibility of repeats. 


20. Closed circuit television and videotapes (including videocassettes and videodiscs) 


1. Adaptable to medium and small au- 
diences. 


1. High initial cost of production equip- 
ment, and requirement of trained staff. 


2. Videotape repeatable to fit learning 
schedules. 


2. Electricity required. Although portable 
equipment works off battery, this needs 
charging from power source. 


3. Film advantages 1, 3 and 4 apply (see 
above). 3. Receivers are expensive and require 


maintenance. 


4. Valuable for magnification of image, 
recording intimate situations, micro-teach- 
ing, recording of developments in clinical 
syndromes or in scientific experiments, 
“bringing the village into the classroom”’, 
recording emergencies, etc. 


5. Portable equipment can function on 
battery for field recording. 


Sound media 


21. Radio broadcasts 


1. Adaptable to large and small audiences 1. Special studio facilities and staff re- 
in widely separated areas. quired for broadcast (less expensive and 
2. Conserves resources of instructors by complex than television). 

broadcasting simultaneously to many 2. Learners must adapt to fixed schedule, 
classes. not other way round. 
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21. Radio broadcasts (continued) 


3. No immediate feedback and no 


audience interaction. 


3. Capable of gaining and maintaining 
attention. 


4. Reception equipment relatively cheap 
and will function on batteries. 


5. If combined with prepared materials 
(radiovision) can be improved learning 
tool. 


22. Sound recordings 


1. Adaptable to any size of audience. 


1. Use for individual learning demands 


2. Especially suited to individual and many playback units. 


small group learning. 2. Good quality recording demands studio 


3. Because of stop and playback facili- facilities. 


ties of tape, can be student paced. 


4. Cheap, battery operated cassette players 
available, and cassettes relatively cheap. 


5. Many uses — to provide sound for slide 
sequences, for micro-teaching, heart sounds, 
for posing problems, etc. 


Note concerning the preparation of “transparencies ’ 
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Certain pages of the Handbook are printed in large type 
and can be used as matrices if you should wish to make 
your own transparencies for projection using an overhead 
projector. 


There are different methods, depending on the photo- 
copying equipment available. 


In general, all you need do is to place a sheet of trans- 
parent, thermosensitive acetate film on top of the Hand- 
book matrix! and put the whole in the photocopying 
machine. In a few seconds you will have a transparency 
ready to be projected on your overhead projector. 


Please follow the instructions applying to the particular 
photocopying machine available to you. 


1 You can obtain them by removing the binding from the Handbook and 


placing the sheets in a ring-file. 
HoooooooOoOoOoOoOoOoOOoOoOOO00000000000 
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self-learning packages 


If you agree that one of your functions as a 
teacher is to prepare learning aids for your 
students and to help them obtain the infor- 
mation they need for independent learning, 
then read the following pages which explain 
what self-learning packages are and give an 
example of this learning technique. 


We have seen that a professional task derived 
from an intermediate objective may serve one 
or more specific objectives, either final or 
enabling. The task must be broken down to 
the level of the specific objective, which must 
have two characteristics: it calls for a single 
activity; and it can be satisfactorily evaluated 
by applying one criterion or a group of criteria 
measuring a given domain. 


Such a specific objective, or task, constitutes 
the basis for constructing ‘“packages’’ 
designed to facilitate self-learning. 


Each self-learning package will comprise: 
1. The statement of the specific objective. 


2. The full documentation needed to achieve 
the objective. This may include references to 
any of the following: existing papers, tape 
recordings, slide sequences, places, persons, 
and so forth. 


3. A formative evaluation instrument to 
enable the student to assess the extent of his 
success in reaching the objective. 


In some cases, several closely linked specific 
objectives are related to the same learning 
activity. It is then advisable to avoid too 
detailed a breakdown, and to combine the 
different ‘‘packages’” for such a learning 
activity into one “‘super-package”’. 


A simple example of a self-learning package is given in the followin 
would also include: slides and transparencies, X 
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A self-learning package is a document containing all that 
is necessary for a student to attain one or more educa- 
tional objectives independently of the teacher. Using 
these packages, the student can take over a large part of 
his training, while the teacher remains available only 
when needed. 
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The technique of self-learning packages is similar to 
that of the Keller plan (see Bibliography), but the latter 
also includes the coordination and supervision of the 
student’s activity by monitors, and group work sessions 
to stimulate students’ motivation. 
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field work, and so forth. 
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g pages. More complex examples 
‘tay photographs, loop films, laboratory and 


self-learning package 


WHO-sponsored Training Course in Epidemiology 


Main topic: Sampling 


Sub-topic: Sampling techniques 


(Prepared by Mr. T. K. Sundaresan) 


Ono COD ee oO) oO: ONO oO Pee ee eee 


Relations to the functions of the epidemiologist. 


Sample surveys are an efficient and economical means 
for obtaining information on matters of epidemiolo- 
gical importance, e.g. prevalence of specific diseases. 
There are several sampling techniques available to suit 
different practical situations and specific objectives of 
the investigation. For a given situation one technique 
can be more efficient and economical than another. A 
knowledge of these techniques, their advantages and 
disadvantages all greatly help in choosing the most 
appropriate technique for that situation. 
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Educational objective (Ref. No. 536 iz 


After studying this document and doing the 
exercises listed at the end you should be able 
to suggest the appropriate sampling technique 
for most epidemiological investigations. 


Pre-requisites to understanding this section 


1. Understanding the concept of random 
sampling (Educ. Obj. No. 5.2)* 


2. Understanding the terms “‘precision”’ and 
“accuracy” (Educ. Obj. No. 5.3)* 


3. Understanding the concept of “confidence 
intervals” (Educ. Obj. No. 5.4)* 


1. Preamble 


The epidemiologist, wishing to carry out a 
sampling investigation, has often to choose one 
of several techniques of sampling. Some of 
the considerations that play a role in the 
choice of the most appropriate technique 
are: availability of a suitable frame, facility 
of the field operations, acceptable precision 
of the estimates, the sub-categories of the 
population for which separate estimates are 
required, etc. There are many techniques 
available but in this package only the following 


will be considered: 

1. Simple random sampling 
2. Stratified sampling 

3. Two-stage sampling, and 
4 


Cluster sampling 


Simple random sampling 


Definition: A sample of size n, drawn from a 
population of size N in such a way that every 
possible sample of size n has the same chance 
of being selected, is called a stmple random 
sample. 


Example: It is desired to determine the pre- 
valence of a certain infectious disease among 
the 1000 people in a village (N = 1000). The 
desired accuracy of the estimate dictates that 
100 people from the village must be examined 
for the disease (n = 100). A list of all inhabi- 
tants is obtained (the population frame). Then, 
using a table of random numbers, a sample of 
100 persons is selected in such a way that each 
person in the village (i.e., each entry in the 
list) has a 10% (n/N) chance of being selected 
for the examination. 


*Note: These references are to a document distributed to participants at the WHO Training Course. 
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Advantages: 


1. Simple design. 


2. Requires no knowledge of the distribution 
of the study variable in the population. 


Disadvantages: 


1. Requires a population frame which can 
be expensive or simply not available. 


2. Travel expenses could be high if the pop- 
ulation is spread over a large area. 


Stratified random sampling 


Definition: A stratified random sample is 
one obtained by first separating the pop- 
ulation into non-overlapping groups, called 
strata, and then selecting a simple random 
sample from each stratum. 


Example: Continuing with the example given 
above for simple random sampling, before 
selecting the random sample of 100 the entire 
population is divided into age-groups (strati- 
fied on age). Then a simple random sample is 
taken from within each age-group (stratum). 
The total number of examinations is still 100, 
but now the prevalence is available by age- 
group and the precision of the overall preval- 
ence estimate is improved. This improved 
precision is due to the fact that sampling is 
now done within more homogeneous groups 
(under the assumption that prevalence is 
correlated with age). 


Advantages: 


1. Estimates are available for each stratum of 
the population and not just one overall 
estimate, as in simple random sampling. 


2. Some gain in the precision of the overall 
population estimate, with an appropriate 
method of stratification. 


Disadvantages: 
1. A population frame is required. 


2. Certain preliminary information is required 
on the variable(s) to be used for stratification. 
This may be expensive or simply not available. 


3. While stratifications on certain variables 
may improve the precision of the estimate of 
one parameter (prevalence of a disease, in the 
above example) it may give rather poor esti- 
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mates for other characteristics to be studied 
at the same time (e.g. fertility). 


Cluster sampling 


Definition: A cluster sample is a simple ran- 
dom sample in which each sampling unit 1s a 
collection, or cluster of elements. 


Example: Continuing with the above example; 
if a population frame is not available, then the 
sampling may be done as follows: a list of 
dwelling units is drawn up; this is much easier 
than a complete list of individuals. A simple 
random sample of dwelling units is selected 
and all individuals in each selected unit are 
examined. 


Advantages: 
1. Reduction in travel costs 


2. Complete population frame not necessary 


Disadvantages: 


1. The variability within and between clusters 
must be considered. If the variability within a 
cluster is low, this could lead to unnecessary 
within-cluster precision, using the same 
number of examinations, while insufficient 
precision is obtained for the overall estimate. 


Two-stage or sub-sampling 


Definition: A two-stage sample is obtained 
by first selecting a sample of groups and then 
selecting a (sub) sample of elements from 
within each of the sampled groups. 


Example: Continuing with the previous 
example; the investigators decide to use the 
list of dwelling units, as the population frame 
is not available, but wish to cover more dwel- 
ling units with the same number of examin- 
ations. A simple random sample of dwelling 
units is selected, then, as the second stage, a 
simple random sample of the individuals 
within each of the selected dwelling units 
is chosen. 


Advantages: 


1. Wider coverage. If it is believed that 
individuals within the same dwelling unit are 
similar with respect to the quantity measured, 
then it makes little sense to measure the same 
thing so many times. 


2. Does not require a complete population 
frame. 


-# 


Reduction of travel costs. 


Disadvantages: 


) 


self-testing exercises 


(a) 


(b) 


It sometimes can cause difficulties to in- 


A survey is planned to determine fertility 
rates in a large geographical area for 
which no accurate lists of individuals 
exist; however the villages and towns are 
well documented. 


1. Suggest two _ possible 


schemes. 


sampling 


2. State two advantages that each has 
over the other. 


It is desired to know the morbidity rate 
among the population of a city. The 
results of a recent census, including age 
and sex, are available. 


answers 


(a) 


1. Cluster and two stage 


2. Cluster over two stage: 


© reduced travel cost (must visit 
fewer villages) 


© simplified design 


O requires less knowledge of the 
distribution of the variables measured 


Two stage over cluster: 


© wider coverage (provides esti- 
mates for more towns and villages) 


© more flexible, one can alter sam- 
pling fractions to attain increased 
economy if there is some preliminary 
knowledge about the variability 
among villages 


clude in the sample only a portion of a house- 


hold or group. 


z 


Incomplete information concerning the 


distribution of the quantities measured at the 
various stages can lead to poor results. 


(c) 


1. Suggest two 


schemes. 


possible sampling 
2. State two advantages that each has 
over the other. 


It is desired to estimate the prevalence of 
tuberculosis in a country. The population 
census is at least 10 years old and it is 
known that movements of population 
may have taken place. It is also suspected 
that the prevalence in urban areas is 
likely to be different from that in rural 
areas. Make any assumptions, relevant to 
your country context, on operational 
factors and suggest a sampling scheme. 


(b) 1. Simple random and stratified random 


(c) 


2. Simple over stratified: 
© simplified design 
© no prior knowledge of the dis- 
tribution is necesssary 
Stratified over simple: 


O estimates by age and/or sex may 
be obtained 


© more precise overall estimates 
may be obtained 


Suppose is it most efficient operat- 
ionally 


1. to examine all members of the house- 
hold once a household is selected, and 
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2. to station the mobile X-ray van and 
the team in one area for at least one 


week 


3. and suppose that 50 individuals can 
be examined in one day, and further 
that 


4. at least 20,000 individuals have to be 
examined to yield an estimate with 
acceptable precision. 


The following can be recommended as one 
possible procedure. 


Because of the operational factors outlined 
it would be best to adopt a multi-stage cluster 
sampling, stratified into rural and urban areas. 
Suppose it is decided to examine 10,000 indi- 
viduals in the rural and 10,000 in the urban 
areas. Taking as an example the rural areas: 
the 10,000 individuals could be considered as 
the population of 40 clusters consisting of 


250 individuals in each cluster (50 individuals 
per day) for five days a week. If the average 
household size is 5 this would mean 50 house- 
holds in each cluster. 


Based on the old census, the country can be 
divided into its major administrative divisions, 
and the 40 clusters allocated to these divisions, 
in proportion to their population Within each 
major administrative division, minor divisions 
where the cluster should be located may be 
selected at random at the second stage. Further 
stages may be introduced if the divisions are 
large. 


Once the last stage, say a sub-division with a 
population of around 1000 is reached, a 
complete census of this sub-division is made 
and a household selected at random. Fifty 
consecutive households in a predetermined 
order will give the required cluster. 


criteria for priorities in 


selecting packages 


1. They should deal with an educational 
problem of major importance for the majority 
of students. 


2. They should be directed to the solution of 
a health problem of major importance for the 
people. 


3. They should make good a deficiency in the 
existing teaching system or in existing educa- 
tional tools. 


4. They should not needlessly be super- 
imposed on another package dealing with the 
same problem. 


5. They should allow for the possibility of 
prompt updating when required. 


6. They should be more effective and econo- 
mical than any other educational tool serving 
the same purpose. 


Educational concepts that are useful in weighing 
the advantages and disadvantages of self-learning packages 


or other teaching methods. 


1. Educational activities should be directly 
_ related to the tasks to be performed by the 
trainees (relevance). 


2. Tasks should be explicitly defined and 
should correspond to community health needs. 


3. Educational activities should be designed 
to meet students’ needs, with emphasis on 
individual training and the possibility for each 
student to work at his own pace, even in mass 
teaching situations. 


4. The educational system should provide 
the student with the work tools he needs for 
his continuing education. 


5. Educational activities should if possible 
take place in the same sociocultural envir- 


onment as that in which the professional tasks 
will be performed. They should include the 
practice of the same skills as will be required 
in real problem-solving situations. 


6. All training activities must be accompanied 
by formative evaluation. 


7. Learning techniques also must undergo 
evaluation during use in order to ensure that 
they really correspond to the students’ needs 
and are in accordance with the sociocultural 
environment; they must also be practicable 
in use for both students and teachers. 


8. The overall evaluation of the programme 
must be incorporated in the system. 


how to use the packages 


Introduction to the use of 


Self-Learning Packages 


The student studies the 
first package when he needs 
to do so to solve a health 
problem that he meets in 
the course of field work 


Help is given by the 
teacher on request 


The student works 
at his own pace 


The student does the 
Formative Self-Testing 
Exercise 


The student takes the 
Certifying Test 


Discussion with teacher on 
the difficulties encountered 


The student goes on to 
study the next package 


some introductory remarks on the 
construction of a student-orientated 


programme 


It has already been said that the educational 
spiral should be taken into account in cons- 
tructing a programme (see p.3.03) 


Other major considerations are: 


The student should be motivated. Learning is 
not only an intellectual but also an emotional 
process. The student’s motivation will be all 
the greater if he takes part in preparing bis 
own learning programme. This is a fundamental 
consideration, and will be dealt with later 
(p. 3.63). 


The student should be able to see the links 
between his different learnmg activities. 
However, the health professions have become 
more and more complex and teaching insti- 
tutions have had to use an ever greater number 
of teachers in a growing variety of subjects. 
The result has been a disintegration of pro- 
grammes. It is therefore important to ensure 
that the programme is not itself constructed 
in such a way as to be a source of disinte- 
gration. The student should be enabled to 
participate in a large number of learning 
activities, either simulated or real, which 


should constitute true exercises in integration. 


The student should be able to connect practice 
with theory. In most teaching institutions, this 
connexion is rarely close. Very often, the 
emphasis on theory in examinations has an 
unbalancing effect. The learning programme 
should be oriented to the real life situations 
that will be met in the course of future pro- 
fessional activity. As he tackles each health 
problem the student should become gradually 
more proficient in analysing a situation, 
gathering the necessary data, perceiving the 
relevant theoretical bases, perhaps through 
the use of self-learning packages, and profiting 
from a self-evaluation test that keeps him 
informed of his progress. 


If the above elements are kept in mind, the 
learning programme will leave room for 
individual training and enable the student to 
develop a method of work that will serve for 
his effective continuing education. 


The role of the teacher is of prime importance 
for the proper implementation of such a pro- 
gramme. It was described on pp. 3.33 - 3.37. 
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The essential point in constructing a student-oriented 
programme is to organise the rotation of students from 
different years in the different places of learning activity 
(professional fields) in small groups under the responsi- 
bility of teacher-coordinators; in that way students are 
progressively confronted with health problems according 
to their level of abilities.* They have at their disposal a 
variety of self-learning packages and other sources of 
information and experience such as workrooms, library 
and laboratories, while their efforts are guided and their 
progress is measured by a system of formative evaluation 
based on pertinent educational objectives. 
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The scheme outlined above is oriented to 
students’ learning needs, and is a far cry from 
a teaching programme conditioned by the 
competency and availability of the teachers. 


After doing some of the exercises in selecting 
teaching methods (pp.3.57 and 3.58), you 


could refresh your memory on integrated 
teaching (pp. 3.59 and 3.60) and also on inte- 
grated learning (p. 3.61), before going on to 
consider how to plan programme reforms 
(pp. 3.66 — 3.74 ) with all that implies in the 
way of constraints and obstacles to innovation 
and change. 


* & more detailed description of the advantages and limitations of this concept, called “community based education”, can be found in 
WHO Technical Report Series No. 746, 1987 (Community based education of health personnel: report of a WHO Study Group). 
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Do the following exercise if you wish to gain practice in selecting appropriate teaching methods 
and/or educational media to help students attain a given educational objective. 


Look at the specific educational objectives on page1.52, or take any other specific objective you 
may prefer, and select the appropriate method(s) or media (indicating the number, from 1 to 20, 
for each given on pp.3.42 — 3.47), and describe the constraints likely to be encountered in your 


institution. 


Objective 
No. 


handouts (6) 
work in small 
groups (2) 


field work (5) 
work in small 
groups (2) 


Constraints to be overcome 


requires the duplicating of patients’ histories, with no 
budget provision up to now for paper or machine; 
peer-teaching is not customary and will need a trial period. 


requires long preparation including the prior selection of — 
families to be visited in collaboration with a social 
worker. 


Check whether your choice is consistent with the educational concepts on p. 3.53. 
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ional objectives on page 1.52, or take any other specific objective you may 

Be ee stbe we akin methods that will enable a class of 200 students of the same 

year to attain the objectives. Compare the two methods using the criteria appearing in the follow- 

ing specification table, according to the following code:* } 

— Effectiveness (the capacity of a teaching method to help a learner to reach a given objective) 
is either high (8), average (6), or low (2). For example, a method is all the more effective if: it 
encourages the student to be active, emphasizes the personal nature of learning, highlights 
self-evaluation, permits confrontation of ideas, facilitates discovery. 


— Cost (the resources expended in carrying outa teaching method; resources /argo sensu: salaries, 
time spent by teachers and students, including transportation time, etc.). Cost may be high (2), 
average (4), or low (5). 

— This code extends from 4 (low effectiveness and high cost) to 40 (high effectiveness and low 
cost). To be acceptable for use a method's index should rate at least 24. 


— Feasibility (see definition under practicability page 2.34) should be considered as “can be done 
immediately” but also “could be done soon”. In the following example the handout may not be 
immediately available but its cost/effectiveness index is high enough to warrant its preparation 
as soon as possible. 


Establish the differential diagnosis 

Objective | ofacaseofanaemia onthe basis 
of the haematological data in the 
patient's records. 


Lecture on the differential diag- 
nosis of anaemias—1 hour 


a Handout giving book references 


on anaemias, patient histories, 
and a self-testing exercise 


Feasibility 


Imme- 
diate 


*This code was prepared in collaboration with T. Freyvogel (Basle) and L. Roy (Geneva). 
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the concept of integrated teaching 


1. Definition 


Integration (from the Latin integer = whole) 
means coordination of different activities to 
ensure harmonious functioning. 


This definition suggests that the result of har- 
monious functioning will be greater efficiency. 
In our materialistic world efficiency is a dogma. 
In the developing countries it is the key to 
survival. Neither the idea nor the word is 
new. At present both are fashionable and, as 
is often the case in such situations, infatuation 
and passion are more in evidence than logical 
analysis and objective appraisal. Such things 
are used, abused, transformed and deformed. 


The password is “‘integration’’. What is inte- 
gration? 


We are concerned here with the meaning of 
integration in the field of professional edu- 
cation and training. To paraphrase the de- 
finition given above, integration in this field 
is “the coordination of different teaching 
activities to ensure the harmonious fonction- 
ing of the educational process for more 
effective health manpower development. 


2. The purpose of integration 


Before the era of the mega-universities with 
thousands of students and before the era of 
specialisations stemming from a technology 
ramifying in all directions, the health pro- 
fessions were learned in the presence of the 
patient and in direct contact with the pro- 
fessor — a natural source of integration. 
During training, there were of course several 
professors and the student — second natural 
source of integration — assimilated, analysed 
and synthesised the information. 


Since the beginning of the twentieth century, 
the university-factory, the numbered and 
card-indexed student, and the super-specialist 
have made their appearance. Fundamental 
and staggering scientific discoveries are being 
made faster than textbooks are revised and 
reissued. Research laboratories are draining 
away and absorbing funds intended for train- 
ing centres. Teachers engage in research since 
their future depends on the results they 


obtain in their field. Without organised teacher- 
training, they are entrusted with students who 
are not supposed to appraise their perform- 
ance. Reputation depends on the number of 
publications rather than the quality of the 
health personnel trained. Theoretically, the 
health worker, once trained, is a complete 
and integrated element. In practice, he is the 
outcome of a non-integrated training lacking 
in human feeling. 


Human feeling, the concept which should 
inspire students and teachers, implies concern 
for improving the health of the people and 
should be the basis of the educational ob- 
jectives of the university centres for health 
sciences. These objectives, shared by teachers 
and students alike, will be more easily attained 
if the various teaching activities function 
harmoniously; hence the concept of integrated 
teaching. 


3. The advantages of integrated teaching and 
its limitations 


The advantages attributed to integrated tea- 
ching are many and attractive. They include 
the synthesised presentation of important 
health problems, the avoidance of contra- 
dictions and pointless repetition, the respect 
for a logical order and the pruning of non- 
essential details, the improvement of the 
quality of teaching and teachers through 
emulation and of the relations between 
departments, and a better utilisation of 
teachers. 


Yet some faculties that have tried it have 
given it up. A much larger number claim to 
use it but, in fact, do not do so or do so 
only partially. 


Why is it then, that in spite of so many advan- 
tages, this method has met with more failures 
than successes after a very large number of 
trials over the last 30 years? 


We do not pretend to give a reasoned answer 
to this question, or statistical data for or 
against the method. The facts can be found 
in the literature dealing with this question. 
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We are deeply convinced, however, that the concept of 
integrated teaching suffers from a basic defect: as its 
name indicates, it is teacher oriented. 
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All educational theories have emphasised the 
efficiency of student-oriented methods. 


If you turn back to the principles stated on 
pages 3.18 — 3.37, you will see that very 
few of them have been really put into practice 
in the various trials of integrated teaching. 
Integrated teaching certainly has advantages 


over any type of non-coordinated teaching of 
separate subjects, but it remains a form of 
traditional teaching in which the teacher is 
predominant, and is less conducive to efficient 
learning than educational activities designed 
to help the student and lead him to achieve 
the necessary integration by his own efforts. 
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A programme composed of such educational activities, 
oriented to the student and not to the teacher, is 
suggested for your attention in the following pages. 


It is the concept of integrated learning. 
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the concept of integrated learning: 


a learner-orientated programme with built-in relevance and based on the acquisition of 


problem-solving of skills. 


In teaching the health sciences, educators 
have, up to the present, more often failed 
than succeeded in incorporating that essential 
quality of any programme — its relevance. 
The relevance of a programme is the extent to 
which it facilitates the acquisiton by the stu- 
dent of all the skills necessary for the practice 
of bis profession, whose purpose is to raise 
community levels of health. 


In chapter 1, the basic elements necessary for 
formulating relevant educational objectives 
were indicated. It was then stressed that the 
health needs of populations, taking into 
account their available resources, should 
serve as a basis for such formulation. A further 
point made was that health planners should 
define the types of health service organisation 
most suitable for solving the health problems 
of a given population, as it is within the 
framework of these services that the individual 
members of the health team will play their 
roles, fulfil their functions and accomplish 
their special tasks. It was also pointed out 
that, in defining these roles, functions and 
tasks, account should be taken, as far as 
possible, of what would be expected of 
members of the health team in their future 
work. 


Consequently prospective definitions are re- 
quired: such. forecasts being dependent on 
the complexity of the skills to be acquired 
and the consequent duration of the training 
programme. For those professions requiring 
high levels of competence (physicians, sanitary 
engineers, etc.) it has not been possible to 
define the tasks they will have to carry out in 
10 years’ time, other than in the form of ass- 
umptions which have variable degrees of 
reliability. Perhaps the developments in the 
science of futurology will ultimately make 
it possible to find solutions for this problem. 


Until this is possible, however, other solutions 
should be envisaged. 


A few years ago Laurence Weed suggested a 
new programme of medical education! which 
attempts to resolve the notorious and difficult 
problem of relevance. The originality of Weed’s 
proposal is that it seems to solve this problem 
while remaining within the strict framework 
of training (refer once more to the diagram 
on page 1.08). Its main point is that the stu- 
dents should acquire a method of working 
and thinking, based on a problem-solving 
approach, that allows them to adapt to a rap- 
idly changing situation while at the same time 
acquiring the skills they need to practise their 
chosen profession. 


A type of educational programme, inspired 
largely by that of Weed, is proposed in the 
following pages. Additionally it comprises an 
orientation closely linked to community rather 
than individual health problems, and it allows 
the student himself to make a free and enlight- 
ened choice of the role he wishes to play in 
society. 


The programme is based on a certain number 
of Weed’s premises which must be stated first 
of all, since any disagreement with these 
premises would probably lead to a rejection 
of the proposed programme. On the other 
hand, the mere fact of agreement with the 
premises would not, for most teachers, be a 
sufficient reason for accepting the new pro- 
gramme, and certainly not for using it to 
replace a traditional programme. This new 
approach is, to say the least, revolutionary; it 
requires replacing a whole programme and not 
merely changing parts of it; or starting a 
separate experimental track.* 


The catch is that, at first sight, the premises 
seem relatively harmless and so inoffensive 
that the most traditional educator is ready 
not only to agree with them but even to 
claim that they are nothing new and that 
he himself adopted them long ago (if, indeed, 
he did not think of them himself). 


‘Described in his book: Medical records, medical education and patient care. Cleveland, The Press of Case Western Reserve Uni- 


versity, 1970. 


?Report of a meeting on “Parallel innovative tracks in established health sciences institutions: a strategy for disseminating change.” 


October 1986, Albuquerque, New Mexico, USA. 
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Weed’s premises 


i 


General educational objectives 


A health professional should be able: 


1.1 to set his personal goals related to 
his work within the organised system of 
health services, and to develop his own 
approaches to the achievement of these 
goals; he must also seek criticism from 
teachers and fellow students. 


1.2 to define health problems and to 
collect relevant data and utilise them for 
solving the problems. 


1.3 to audit his own performance or 
that of a colleague using carefully 
established criteria. 


1.4 to demonstrate the qualities of pro- 
fessional thoroughness, reliability, sense 
of analysis and efficiency. 


Teaching and learning techniques 


The programme must make it possible for 
the student: 


2.1 to acquire basic skills, and not 
merely basic knowledge; practical and 
communication skills and not only 
intellectual ones; the preponderant role of 
recall must be reconsidered, the lasting 
ability to recall must be based on repeated 
use. 


2.2 to learn and think independently 
and to learn from his experience. Every 
patient will be a source of data, while 
the teachers should ensure intellectual 
discipline and encourage value judgements. 


2.3 to acquire independently, by the 
study of documents and_ information 
prepared for the purpose (books, scientific 
journals, duplicated texts, audiovisual 
aids, etc.), everything that is usually 
transmitted by traditional lecture-courses, 
provided that the educational objectives 
are known to him and that he performs 
formative evaluation exercises constructed 
on the basis of the objectives. 


2.4 to acquire his professional skills 
through the practice of real professional 
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acts of increasing difficulty, compatible 
with his level of achievement. 


2.5 to practise professional acts under a 
control and supervision that leads him to 
understand the underlying basic principles. 


2.6 to find solutions to a diversity of 
new problems, thus developing his ability 
to carry on by himself once the problem 
to be solved has been defined. 


In the organisation of studies the following 
points should be considered: 


3.1 The quality of achievement and 
acceptable performance for a given task 
should be regarded as a constant. What 
can vary is the number of types of tasks 
that a given health professional should be 
able to accomplish. 


3.2 The length of study programmes 
should not be regarded as a constant. It is 
preferable to have a clear definition of the 
skills to be acquired, and for the time 
necessary to master them to vary for 
different students, depending on their 
ability and drive and on the availability 
of teachers and educational facilities. 


3.3 There is no natural frontier in the 
practice of health care between the 
physical sciences and the humanities, nor 
between science and culture in the 
broadest sense of the word. Students 
should be enabled to define the patient’s 
social and psychological problems in 
addition to his ‘‘medical’ ones, and to 
look for a logical solution, either by them- 
selves or with the aid of specialists, for 
each of such problems. Simply to organise 
courses in the social and _ behavioural 
sciences does not automatically ensure 
that future physicians will be able to 
relate their daily practice to the culture 
to which they belong. 


3.4 The main educative role of the basic 
scientist is not to train health personnel 
in his special subject but to audit their 
performance to ensure the correct applic- 
ation, or to detect any violation, of the 
basic scientific principles of that subject. 


competency-based programme/contract 
an excursion into Utopia 


The programme includes four phases at the 
end of which the student is awarded a degree 
certifying be bas reached a satisfactory level 
for certain specified skills. 


The total duration of each phase may vary for 
each student and is determined by the achieve- 
ment of an acceptable level of performance 
for a given number of educational objectives 
which together constitute the basis of a con- 
tract between the student and the teaching 
institution. Depending on the terms of the 
contract, the degree may be that of doctor of 
medicine, nurse, medical assistant or, for 
example, specialist in cardiology or in stoma- 
tology. 


Description of the Programme/Contract 


First phase (very approximate length — one to 
three months). Goal: to learn bow to learn. 


1. At the end of the first phase, the student 
should be able to: 


1.1 show competence in planning educ- 
ation at initiation level; 


1.1.1 define educational objectives 
on the basis of tasks; 


1.1.2 plan alearning programme, and 


1.1.3 construct tests and other for- 
mative measuring systems; 


1.2 utilise the resources of the institution, 
such as microscopes, slides, library, 
computers, or members of the health 
team; and 


1.3 describe the concept of the (health) 
team approach. 


Second phase (approximate length 12 — 18 
months). Goal: to identify his professional 
role in society. 


2. At the end of the second phase, the stu- 


dent should be able to: 


2.1 prepare a list of what he considers 
the ten main health problems of the 
community ; 


2.2 collect a data base on the community 
and interpret these data in order to: 


2.3 identify its priority health problems; 


2.4 describe the health activities (services) 
which would enable the community 
health problems to be solved, and 
compare these activities with those 
usually undertaken ; 


2.5 identify the professional role which 
he (the student) wishes to assume at 
the end of his training; 


2.6 describe the functions, activities and 
tasks corresponding to that role; and 


2.7 prepare a set of educational object- 
ives! (with evaluation criteria) and 
indicate the acceptable level of per- 
formance for each of such tasks. 


At the end of the second phase, a contract is 
signed between the student and the teachers 
(Faculty, School or other institution). The 
contract provides, on the one hand, that the 
student undertakes to achieve the educational 
objectives (described under 2.7 above) within 
a given period (e.g. three, four or five years), 
and, on the other hand, that the teachers 
undertake to facilitate the student’s work 
(see the teacher’s functions, pp.3.33 — 3.37). 
It is understood that the teaching staff has 
full responsibility for certifying the abilities 
of the student before he is authorised to prac- 
tise without supervision. The contract may be 
revised from time to time if the results of the 
evaluation procedures justify that course. 


Once the contract is signed, the student really 
begins to learn his profession . 
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! It should also be borne in mind that the definition, at 
the most general level, of educational objectives for the 
higher levels of professional staff must include the 
following objective: ‘To be capable of finding solutions 
to problems arising from new situations’’. From that 
general objective, a whole series of more specific ob- 


jectives will be derived, including the acquisition of a 
scientific manner of thinking, for which the teaching of 
the basic sciences will have a large responsibility, and of 
a method of work essential for the continuing updating 
of professional skills. 
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Third phase ' (very approximate length — 1 
year). Goal: to learn basic techniques. 


3. At the end of the third phase, the student 
should be able to: 


31 


shed 


cee 


collect the data base on the members 
of the community, both sick and 
healthy (the basic data, such as chief 
complaint, patient profile, systems 
review, physical examination, lab- 
oratory data, etc.) are clearly defined 
by Weed; the reader is advised to 
consult his works and make sure that 
students collect all the necessary 
data); 


audit the data base for a given patient 
collected by another student; and 


carry out clinical techniques (such as 
phlebotomy, sutures, lumbar punc- 
ture, funduscopy or auscultation) 
corresponding to the educational 
objectives described under 2.7. 


Fourth phase (very approximate length — 3 
years). Goal: to acquire professional skills. 


4. At the end of the fourth phase, the student 
should be able to: 


carry out (together with other members 
of the health team) all tasks involved in the 
health promotion/rehabilitation of the 
community in which he is living, and per- 
form the role he has chosen for himself 
(see above 2.5 to 2.7). 


Throughout these four phases, a series of for- 
mative tests are provided to enable the student 
to form an idea of what remains for him to 
do to attain the objectives he has set himself. 
The degree, or authorisation to practise his 
profession, is awarded by the teaching insti- 
tution only when the student has demon- 
strated, by passing the certification tests, that 
he is capable of an acceptable level of perfor- 
mance for the objectives of the contract. 


‘The third and fourth phases correspond to the programme in the form proposed by L. Weed in The New Curriculum, 
the problem-oriented system, Medcom 1972, pp.95 — 104. 
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Unfortunately, it very often happens that, when the 
reader reaches this point, he decides that the programme 
is too utopian ever to be put into practice without a 
cultural revolution. He is therefore urged to reread care- 
fully the brief description of the various functions of 
the teacher given on pages 3.33 — 3.37 which are much 
more attainable within the limits of a certain evolution. 
He should then do the exercise on page 3.39. Perhaps 
a little day-dreaming was not such a bad idea, it can be 
valuable to set oneself an ideal. 


Some universities and schools are already applying a 
good number of these principles. In WHO Public Health 
Papers No. 70 (1978) and No. 71 (1980) you will find 
descriptions of the programmes of the McMaster Univer- 
sity, Ontario, Canada, the Maastricht University, 
Netherlands, the University of Newcastle, N.S.W., 
Australia, or the Xochimilco University, Mexico, a des- 
cription of the nursing training programme at the 
Mahidol University, Thailand, and accounts of training 
programmes for other health workers at Madang, Papua 
New Guinea. 
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briefly then, this programme 


ta is oriented to the needs of society 


= ensures the continuous supervision of the student and 
protects society against incompetent health personnel 


a provides a basis for continuing training 


@ concentrates on the interests of the student and 
encourages individual learning 


8 allows each student to work at his own level and at his 
own pace within a prescribed period 


= @~=6 keeps students in an active situation and gives a leading 
role to experience 


= leads the student to discover and understand the “‘why” 
and “‘wherefore”’ of his learning experience 


= accepts the idea that it is all right to be different 


# increases the student’s sense of responsibility and his 
control over his own learning 


= encourages students to collect, analyse, evaluate and 
interpret data and solve problems 


= emphasizes the importance of cooperative self-evaluation 
= confronts students with real-life situations 

= introduces students to team work 

xu allows students to practise what they have learnt 


% enables students to acquire their professional skills under 
conditions similar to those in which they will later 
practise (community-based education) 
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planning the changes required to 
bring about programme reform’ 


A key component of educational reform and 
innovation is the organisational modification 
needed to plan and implement the desired 
changes. In other words, how will the organ- 
isation be modified to facilitate change? Who 
is to do what and in what sequence? Is the 
completion of some tasks prerequisite to 
beginning others? What supporting actions 
need to be taken and what resources need to 
be made available in order for the change to 
take place in an orderly and effective fashion? 


In order to illustrate some of the planning 
principles which need attention, the attached 
planning schedule has been prepared as an 
example. 


Why a planning schedule 


It is designed to help to answer the following 
questions: 


= If a curriculum is designed on the basis 
of professional skills to be acquired or 
priority health problems to be solved 
rather than time distribution, how should 
this affect the planning process? 


= How can a faculty be organised with a 
view to introducing a “‘new”’ programme? 


Planning schedule concepts 


The concepts both explicit and implied in this 
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plan represent a radical departure from tradit- 
ional practice in schools of health sciences. If 
a school is to shift to the type of system de- 
scribed here, the transition must be regarded 
as a long-term (perbaps four or fre years) 
project. This suggests that: 


= A plan for phasing from the old system 
to the new one should be developed. 


= Aseries of intermediate objectives should 
be set with time deadlines. 


= The objectives should indicate which 
parts of the old system should be im- 
proved, which elements of the new sys- 
tem should be introduced, and in what 
sequence. 


= The plan should be discussed with the 
teachers and appropriately revised before 
becoming institutional policy. 


Sequence of steps (see pages 3.67 — 3.70). 


The system proposed is hypothetical and 
should be used only as a means for beginning 
work on a system appropriate for any given 
school. 


*Adapted from a document by Dr. J. Hess, Wayne State 
University School of Medicine, Detroit, Michigan, USA, 
(1971) 


A representative sequence of steps for planning and implementing a “‘new”’ programme 


Faculty Council University adminis- 


tration; major power 
Programme 
Committee : 


1. Set up a working group (Programme 
Committee) with authority and 


responsibility to plan and co- centres in the Faculty 
ordinate a new programme. or School. 

Members should be selected for 

creative abilities as well as for 

distribution among disciplines. 


In addition, students, practitioners 
and health administrators (central 
level) should be included. 


_ Prepare a written statement of 
overall objectives in terms of the 
functions that graduates should 
be able to perform, and a state- 
ment of the general philosophy 


of the programme. 
Obtain approval of the written Approval to be given University authorities; 
overall objectives and the pro- jointly by the Faculty | major power centres in 
gramme philosophy. Council and the the School; Ministry of 
Revise as needed to obtain agree- Programme Health 

Committee 


ment with the plan in principle. 


_ Select the organizing principles that 

will be used for planning the new 

programme. Examples are: 

— competency base; 

~ body systems in health and 
disease; 

— problem-solving base; 

~ the processes of “clinical” work 

— apreventive model; 


~ community problem orientation; 


~ programme with contract; 


community-based education. 
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Taken by Obtain agreement from 


5. Lay out a general sequence of pro- Programme Faculty Council 


gramme phases in block diagrams Committee 
or outline form based on: 


5.1 Organising principles and 
philosophy of the programme 


5.2 Assumptions and data con- 
cerning capabilities of 
incoming students; 


Terminal programme 
objectives ; 


Available teaching resources. 


Select the disciplines that should Programme 
have major representation in each Committee 
programme phase. 


Organise committees of teachers Programme Faculty Council 
for the main programme phases. Committee Heads of Departments 


Critical factors: 


7.1 Those responsible for each 
phase should: 


(a) be able to advance | 
convincing argu- 
ments in support 
of the programme 
philosophy, and 


(b) have demonstrated 
ability to get things 


done in committee. 


Most committee members 
must be open-minded and 
ready to break with 
tradition. 


Major disciplines mentioned 
in step 6 must be represented. 


1 ° 
Select teachers’ to lead student Committee of Heads of Departments 
groups. Teachers 


1 “é ”” 2 
The term “teacher” should be understood in its new sense, which includes senior students 
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9. Begin training sessions for comm- Programme = 


ittee members and the teacher- Committee with 
leaders of student groups. assistance from 

Heads of the Depart- 
ments concerned 


education consult- 


General subjects: 
ants from: 


9.1 Educational philosophy and 
objectives; 


(a) the University 
itself; and 


9.2 Teaching concepts and 
methodology; and 


(b) other organ- 
isations. 


9.3 Fundamental principles 
essential for the success of 
the teaching programme. 
Evaluation process 


Committee of 
Teachers following 
guidelines from the 
Programme 
Committee and educ- 
ation specialists. 


The requisite level (for students) 
and the intermediate educational 
objectives for each phase are pre- 
pared or reviewed by the 

Committee of Teachers. 


Preferred sequence: Begin with 
last phase prior to graduation 

and work back to entry point of 
programme. 


Teachers of the 
corresponding 
programme phase. 


The objectives or plans for each pro- 
gramme phase are revised as 
necessary. 


Teachers of the 
programme phase; 
Programme 

Committee 


Specific objectives are prepared and 
reviewed. 


Teachers of the 
phase with the help 
of the Committees 
of Teachers and 
education specialists 


Programme Committee 
and Heads of the 
Departments concerned 


Teaching and evaluation methods 
(immediate and long-term) are 
selected and the required adminis- 
trative support is planned and 
organised. 


Teachers of the phase | Heads of Departments 
with the help of the 
Programme 


Committee 


Teaching and evaluation methods 
are implemented. 
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Committee of 
Teachers and 
education 
specialists 


15. Evaluation results are collected. 
Data pertaining to individual 
student evaluation and evaluation 
of the teaching given are handled 
and interpreted separately through 
appropriate channels. 


. Recommendations for impro- 
ving the programme are 
developed. They are based on 
the evaluation data and other 
sources of information and 
advice. 


. The necessary changes are 
implemented. 


qualities of a 


Programme Appropriate individuals 
Committee, and groups. 

Committee of 

Teachers, 


School 


Administration 


Programme Faculty Council, 
Committee, Heads of Departments 
Committee of 

Teachers. 


programme committee coordinator 


He need not necessarily always have all the 
following qualities, but if you had to choose 
at some time or another between two collea- 
gues it is suggested that you choose the one 
most resembling this description. 


He should be: 
@ available 


os motivated 


= have the support of the school admini- 
stration 


= accepted by the students 


and also have as many as possible of the 
following qualities: 


Vast knowledge and wide professional 
experience 


Recognised skill in educational planning 
The personality of a leader 

A reputation as a research worker 
Years of experience as a faculty member 


A reputation as an author 


PS. IF you Knous Suth a person 


please. tele phone immediately 


to W.H.0. Thank yor. 


advantages and limitations 
of a systematic approach 


Possible advantages of a systematic approach 


ik 


It provides a mechanism making possible 
a study planned and organised in advance. 


When it becomes really operational it 
should be more effective than the trad- 
itional approach since it can be improved 
by the feed-back. 


2.1 If it is well organised, it may permit 
greater flexibility in organising the 
use of time of students and teachers. 


2.2 Allowance can be made for certain 
individual variations in students’ 
rate and mode of progress through- 
out the teaching programme. 


2.3 It may prove possible to train more 
students at the same cost if the 
system is carefully structured. 


Possible disadvantages 


Bick 


5 


Teachers must spend more time on plan- 
ning and evaluation, especially during 
the first years following the adoption of 
the system. In many respects it is more 
expensive at the outset to try to con- 
struct a new system rather than to keep 
to the old one. 


It seems clear that a nucleus of teachers 
will have to spend a greater part of their 
time in planning, managing and evaluating 


the programme. These activities could 
be organised using a rotation system 
covering three or four years, so as to 
distribute the work and give teachers an 
Opportunity to become familiar with the 
teaching methodology. 


Necessary supporting elements 


4.1 


4.2 


4.3 


4.4 


A school administration and depart- 
mental heads willing to delegate some of 
their responsibility and authority to a 
new programme planning and manage- 
ment system. 


Budgeting of funds in a manner consis- 
tent with the teaching programme and 
the distribution of decision-making 
authority. 


A review and, where necessary, a revision 
of academic regulations concerning the 
hierarchical system in operation and the 
promotion of teaching staff so that 
teachers’ contributions in planning and 
implementing the teaching system will 
be duly recognised when decisions are 
taken on salary increments and academic 
rewards. 


A faculty council and a programme 
committee that are skilled in arranging 
to co-opt, ‘neutralise’ or ‘remove”’ 
(by promoting, but elsewhere) key 
opponents to the plan. 
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To train large numbers without loss of educational 
quality 


Promote and develop the use of self-learning 
methods and material 


and 


have students share teaching responsibilities* 


The key to effective learning is to adapt learning 
activities to the needs of individual learners 


*Cf. Students Learning From Students. WHO Document 
WHO/EDUC/80/181 


Pa? 


specification tables 


A specification table, or double-entry table, is a kind of check list enabling you, when you prepare 
a programme or an examination, for example, to make sure that the appropriate parameters 
have been taken into account. It sometimes happens that even the best-intentioned teachers who 
have made an effort to define educational objectives go on to prepare a programme or an exam- 
ination which, when analysed, appears to neglect them completely. In order to ‘‘force”’ yourself 
to bear these objectives in mind, you would be well advised to make use of specification tables: 
they will help you to relate each part of the programme directly to a given objective. In other 
words, “‘every part of a programme or an examination should correspond to an objective’, and 
“every objective should have a corresponding element in the programme or examination”. That 
is the price of relevance. 


How to use a Specification Table 


One example of a specification table (p. 1.27) has enabled you to have a brief overview of both 
a teaching programme and a series of examinations. 


A second example (p. 2.38) showed you how to compare different evaluation methods using 
selected criteria. 


A third example (in this chapter, p. 3.58) has helped you to compare two teaching methods, 
using other selected criteria. 


A fourth example will be given in the next chapter (pp. 4.54 — 4.57). By means of another exer- 
cise, you will be able to make a qualitative analysis of a series of tests (the exercises in the Hand- 
book). 


These four exercises will have allowed you to see what an extremely useful tool a specification 
table may be in educational decision-making. 


Remember to construct one whenever that seems advisable. 


Example of organizational chart of an integrated programme planning mechanism 


Educational 
Consultants 


Programme 
Coordinator 


Programme 
Committee 


mm Evaluation Committee 


Year 1 Year 2 Year 3 Year N 


Coordinator Coordinator Coordinator Coordinator 
and Committee and Committee and Committee and Committee 


Teaching Units 
(small groups of students with a tutor) 


Key: Solid lines represent direct lines of administrative decision-making 
Dashed lines represent advisory functions 
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obstacles to change 


es Dynamic conservatism 


i Institutional bureaucracy’s built-in resistance to 
change 


ad Complexity of programme change 
i Lack of institutional resources 


a Lack of model upon which to base changes 


J. Bryant 


iy 


x 
= we j 


Personal Notes By pers) be i Cai i 
bes ta a hs aa tae migive | meinen nts ‘ad : 


‘Sains in chart form the type of Organisation (commissions, committees, councils etc., with a 
ist of the functions of each) which you think could be set up in the institution where you are 


teaching with a view to introducing (or im i 
; proving) a relevant programme. 
with that on page 3.74. ae vegdaade tastes 


Ch ere be Chl) 0) Cr tie ta OF Oe Os ett 


The classically rigid, inflexible departmentalization of 
faculties of medicine is probably the greatest single 
obstacle to educational reform. 


D. Maddison 
Dean, Fac. of Med. 
University of Newcastle, N.S.W., Australia 


a Lear voc eur ole Gly Gh.bP xa) .f) OY tke i), Dey DP De 
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Describe the obstacles you are likely to meet in setting up the type of organisation that you de- 
scribed on the preceding page, and the tactics you could use to overcome them. 


BS O°-Oh oS GY OO Be Do Bo BB neo 


An educational system will never achieve its full 
potential in the preparation of health professionals if 
it continues to guard jealously individual and depart- 
mental autonomy and to maintain a lofty aloofness 
from the health care system. 


G.E. Miller 1977 
NOOO O000OOOOnDoOoooOnooo Og 
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(Check your answers on page 3.84) 
Important: Se/ect only one reply to each question. 


Question 1 


According to present educational trends, all the following statements are true, except one. 
Which one is false? 


A. Asystematic approach facilitates the choice of learning situations and evaluation 
methods. 


B. The modern trend in educational practice is to transfer part of the teacher’s role to the 
student. 


C. Present educational trends seem to lay more stress on the student than on the teacher. 


D. The student’s judgment of the value of an educational programme is perhaps more valid 
than that of the teachers. 7 


E. The primary role of the teacher is to transmit knowledge in the field of his specialty. 


Question 2 


Learning is made easier by an educational situation in which the teacher does all the following 
_ things except one. Which is inappropriate? 


A. The teacher encourages the student to be active. 


B. The teacher helps the student to discover concepts. 

C. The teacher makes frequent checks of the student's level. 

D. The teacher encourages individual learning. 

E. The teacher gets the student to participate in educational decision-making. 
Question 3 


The many definitions of learning include all the following characteristics except one. State 
which. 


A. Amore or less permanent change occurs in the student's behaviour. 
It is not directly observable. 
Repetition by the student is a favou rable factor. 


The role of the teacher is primordial. 


RS Se swe 


Motivation is an essential element. 


py A 


Questions 4 to 7 | 
Indicate for each of the following questions whether reference is made to: 
A. an epidiascope 

B. an overhead projector 


C. aslide projector 


Question 4 
Can project outlined figures (or letters) on a transparent support (25 x 25 cm) so that they can 


be read in a lighted room. 


Question 5 
Can project outlined figures (or letters) on opaque paper so that they can be read in a darkened 
room. 


Question 6 
ls also called a diascope. 


Question 7 


Can project outlined figures (or letters) on a transparent support (50 x 50 mm; 24 x 36 mm) 
so that they can be read in a darkened room. 


Question 8 
A self-learning package must include at least three of the following elements. Indicate which. 


1. A short statement of the subject. 


2. The statement of the educational objectives. 
3. A definition of the required level. 
4. The full documentation needed. 

9. Diagrams, photographs or slides. 

6. A formative measuring device. 
Reply: 

A if the elements are 1,2and6 

ime eae s OS Ore 1,3 and 5 

Rian were os a ee 2,4 and 6 

DIN SHE OS eos, 3,4and 5 

pS Ses a any other choice 


3.80 


Question 9 


Upon which of the following criteria for the choice of teaching material does the quality of 
relevance depend? 


The material should: 

A. be appropriate to educational objectives: 
promote active student involvement; 

be appropriate to the students; 


be of good technical quality; 


moo wo 


be very cheap. 


Questions 10 to 16 
If a lecture course is denoted by X, and a self-learning package by Y, 


Reply: 

A if the method described is consistent only with X 
Be a y 

re rete Ree epi aia ¢ oeoec with both X and Y 
2 | SESS MSG RAY SIS ec ae Se nn with neither X nor Y 
Question 10 


A method which places the student in an active situation. 


Question 11 
A method which allows the student to attain a given educational objective. 


Question 12 
A method which provides the student with a feed-back of information on his progress. 


Question 13 
A method which requires the student to work in a team. 


Question 14 | 
A method which allows the student to work at his own pace. 


Question 15 
A method which appears to be economical in time and in means. 


Question 16 
A method which emphasises self-measurement. 


3.81 


Question 17 ; 
Among the educational objectives to be reached by a class of 250 students of the same year Is the 


following: “to be able to measure the height when lying down of a new-born baby using a portable 
scale”. Criteria: maximum error of + 0.5 cm in 90% of measurements. 


t each student has to spend eight weeks in a MCH (Maternal and Child Health) 


Supposing tha ; at . 
Set order of decreasing efficiency the following four 


centre in the course of his training, /ist in 
teaching methods: 


A 30-minute lecture to the 250 students in a lecture room, followed by a demonstration by 


1 
the instructor. 

2. A 30-minute lecture followed by practical exercises of the technique by the students in 
groups of 25, each group spending two hours with the instructor in rotation. 

3. A demonstration of the technique by an instructor followed by practical exercises by the 


students in groups of 25, each group spending two hours with the instructor in rotation. 


4. Each student receives a handout with diagrams during the training period; valid certifying 
tests are checked by the instructor on request by the student. 


Reply: 

A if the order of decreasing efficiency Is 1234 
SO EAS) (AS OS DR ae Sing. ae 2413 
Od SESS Pa ed re eS 3142 
DD csi Cae Sa Ns pai eae mangers ee AS 2:1 
Question 18 


When preparing a part of the programme whose aim is to stimulate the student's originality in 
looking for solutions to problems, the most suitable group of learning activities is: 


A. The students participate in courses and demonstrations by teachers with a very creative 
attitude who are themselves excellent research workers. 


B. The students participate in laboratory and field exercises during which they reproduce 
basic experiments. 


C. The students participate in a series of hospital, field and laboratory activities, and must 
present the solutions arrived at in clear terms. 


D. The students participate in hospital, field and laboratory activities and must write a report 
describing the experiments in which they have participated. 
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Question 19 


According to L. Weed, all the following statements are true except one. Which is untrue? 


A. Length of training should be fixed in advance for a given group of students depending on 
the type of degree. 


B. The student should be able to define his own work objectives within an organised system 
of medical care. 


C. Anunderstanding of basic principles should result from the students’ confrontation with 
practical problems. 


D. The lasting ability to recall depends for the most part on repeated use. 


Question 20 
A programme specification table (double-entry table): 


A. sa table giving details of a teaching programme (time-table, premises, etc.) 
B. isa list of contents giving detailed instructions concerning programmed learning. 
C.  Isacontrol sheet for checking the specificity of the educational objectives. 


D. Isacontrol sheet for checking whether the elements of the programme are related to the 
educational objectives. 
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Suggested answers for the exercise on pages 3.77 — 3.83 


If you did not give the right answer, reread the following 


Suggested Answers pages. 


3.05 — 3.12, 3.18 and 3.19 


= 


3.25 — 3.31 


3.42 — 3.47 


3.48 — 3.53 
3103 


Or CO OO OF 2 FB GS KS 


— 
oO 


— 
NO 


3.41 — 3.53 


3.41 — 3.59 


3.62 — 3.66 


E 
C 
D 
B 
A 
B 
C 
C 
A 
B 
11 C 
B 
D 
B 
A 
B 
D 
C 
A 
D 
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Congratulations! You have attempted to work out 
a programme corresponding to your subject. 


But be careful 


= Make sure once more that it agrees with the 
professional functions and tasks 


= Pay unfailing attention to relevance 


m Effective teaching can be more dangerous than 
no teaching at all if it is not really relevant 


3.85 


~ 


y GOP AS a te eee, jan ih on i 


7 i ian ie Te . * i a i ° 
: | te Sie ep ee 
i “to iow ot botqmetts svattue¥ anoteluisinnels 3 ma 
| b .. “fovidue 190y Of enibnogesrros Smirnngorg 2 i 


z 


i—e a =e 
7 ve 
oh GS 
417% 

a Wee 


x, luterso ad 3 


Sait aie fF 29m ys 1 terfe1om sono stu onlet 
| a. 2 | ‘fy se pleat bis enoitony? tsriciezator 


7 >) 
.. . i 
r y Zz 


; . a | | Sati iy aa 
_sonvale or noidnetis gnilietny ved. ~~ a 


i ake , 

r ; oe / 7 . oe - _ a ed 
= , : - = ‘ ¥ ‘ iv 

ca ol - - 

> oan ° P 


BT ABOTSNSL Sore ad AS9 pninosat evitostia oe 
a1 1 Vilees fon.2at 27 Ti le ts priinogel On a 


my | Si as 

panes ited. j 

ef 12 ttBVaie 

ee BY. Sp Thld 

a : Ra a a ' 


7 


I 


i 
a 3 
> > 


+ 
¥ 


Le 
Te 


aah 
r] _ 7 ad 
- - ro, 
2 ae . . 
iis de 4 


test and 
measurement 
techniques 


the practice of test and 
measurement techniques 


This fourth chapter reviews a certain number of measuring instruments, their advantages me 
limitations, and how they should be used. It shows the wide range of such instruments a 
the importance of choosing them in accordance with the educational objectives to be measured. 
It is stressed that every teacher should make a certain minimum of statistical calculations when 


he carries out an item analysis of a test. 


Those with a deeper interest in these problems are strongly advised to consult the following 


publications: 


Public Health Papers - WHO 


= No. 36 Areview of the nature and uses of examinations in medical education, 1968. 


= No.52 Development of educational programmes for the health professions, 1973. 


= No. 72 Assessing health workers’ performance: A manual for training and supervision, 


1980. 


After having studied this chapter and the references indicated, you should be able to: 


1. Indicate the different elements that 
should be considered in the evaluation of a 
teaching programme. 


2. Indicate the different elements that 
should be considered in the evaluation of the 
educational objectives of a teaching pro- 
gramme. 


3. Define the advantages and limitations of 
a system of evaluation of teaching by the 
students. 


4. Construct an observational rating scale 
and/or a practical test to evaluate the behaviour 
of a student in the domain of communication 
and/or practical skills. 


5. Propose a question for a written (open- 
book) examination of the ‘‘essay’’ type or a 
series of six short, open-answer questions and 
indicate the norms of performance permitting 
objective marking (marking table). 


6. Draw up three multiple choice questions 
(MCQ) in the domain of intellectual skills — 
at least two of the objectives must measure an 
intellectual process superior to level 1 ‘“‘simple 
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recall’’ (either level 2 ‘‘interpretation of data”’ 
or level 3 “‘problem-solving’’). 


7. Indicate the advantages and limitations 
of a programmed examination. 


8. Define the following terms: prerequisite 
level test, pre-test, interval test, comprehensive, 
pre-final; indicate their purpose and the stages 
at which they are set. 


9. Explain the difference between a relative 
and an absolute criteria test. 


10. Calculate the acceptable pass level for a 
MCQ examination and establish the scoring 
criteria and norms which permit determin- 
ation of the passing grade of a mini-test (made 


up of the questions mentioned in objectives 
5 and 6). 


11. Do an item analysis of a question (cal- 
culate the difficulty index and the discrimin- 
ation index) and draw the relevant conclusions. 


12. Construct a specification table for an 
examination. 


the educational spiral 


Defining 
Tasks and 
Educational Objectives 


Implementing ALG errr ES 
Evaluation 


Preparing and 


Implementing an 
Educational Programme 


Planning 


an Evaluation 
System 
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why evaluate? 


Society, which finally pays the bill for health 
activities, relies on us to train health personnel 
to perform a social function corresponding to 
the health needs and demands of the commun- 


ity they are to serve. 


Training centres for health personnel form an 
integral part of society, and must be prepared 
to operate within and for that society. 


Health activities are of a nature to stimulate 
social awareness and to provide leverage for 
social development. The following are some of 
the questions that must be asked in any 
attempt to discover whether training centres 
go to sufficient lengths in preparing the diff- 
erent kinds of health personnel and giving 
them a training that is “‘pitched”’ to meet the 
health needs of our societies. 


= Do the graduates think and behave in 
terms of health rather than of disease? 
That is to say, do they apply techniques 
of prevention and health promotion and 
not only those of cure and rehabilitation? 


= Do the graduates think and behave in 
terms of family and community, rather 
than in terms of the individual sick 
patient? 


= Do the graduates think and behave in 
terms of membership of a health team 
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consisting of doctors, nurses and other 
health workers as well as social scientists 
and others? 


= Do the graduates think and behave in 
terms of making the best and most 
effective use of the financial and material 
resources available? 


= Do the graduates think and behave in 
terms of their country’s patterns of health 
and disease, and the relevant priorities? 


Consider the institution where you are 
working — and reflect on the above ques- 


tions. If you can reply “yes” to all of them, 
then the essential has been achieved. 


If, on the other hand, you cannot unreservedly 
say ‘‘yes’’ to certain of them, it is urgent for 
you to reconsider the orientation and the 
training programme of the school concerned. 


To that end, several steps can be suggested: 
1. Evaluate the programme as a whole. 


2. Evaluate the general and intermediate 
educational objectives. 


3. Have the students evaluate certain aspects 
of the curriculum. 


4. Evaluate the students’ level of perfor- 
mance. 


guidelines for evaluating a health 
personnel training programme 
— summary description 


The following guidelines are meant to be both 
general and comprehensive. The evaluator must 
adopt a realistic approach and decide in each 
case what is essential in any particular sit- 
uation, 


The guidelines take the form of a series of 
operations but, obviously, in practice these 
will often be carried out in a different order. 
What is essential, in any evaluation study, is 
for the evaluator to be constantly re-examin- 
ing the information obtained, reformulating 
his questions and, even more important, re- 
vising his judgments. He is thus often obliged 


Guidelines 
Phase I — Orientation 


1. Determine the general characteristics of 
the teaching programme. 


2. Ascertain the general characteristics of 
the administrators, teachers and students. 


3. Determine the financial resources avail- 
able for the programme and its physical 
facilities. 


4. Clarify the aims of the proposed eval- 
uation. 


5. Make an inventory of what information 
is readily available about the teaching pro- 
gramme. 


6. Determine the resources available for the 
evaluation and fix a time limit for its com- 
pletion. 


7. Clarify the evaluator’s role and that of 
the other persons taking part in the evaluation. 


8. Makeapreliminary appraisal of the nature 
and the feasibility of the proposed evaluation. 
Phase II — Design of the Evaluation 


9. List the questions and issues to be con- 
sidered. 


to retrace his steps and, ideally, should review 
his whole orientation. 


The guidelines are presented under four 
main headings corresponding to the four 
phases of the evaluation: 


I Orientation 
II Design of the evaluation 


III Gathering information on the programme 
and its effects 


IV Analysis and reporting 


10. Determine the appropriate sources of 
information and the procedures for its col- 
lection. 


11. Design the evaluation. 

12. Draw up an evaluation schedule. 

13. Draw up an evaluation budget. 

14. Obtain feedback check on the evaluation 
design, schedule and budget. 

Phase III — Gathering information on the 
programme and its effects 

(a) Context and objectives 


15. Describe briefly the context in which the 
programme operates. 


16. Define the objectives by drawing up a 

list 

(i) of the future functions, taking account 
of the main health problems and the 
tasks and responsibilities deriving from 
each, and 


(ii) of the training objectives, taking account 
of the principal tasks the student should 
be able to perform satisfactorily at the 
end of his training. 
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(b) The characteristics of the students on 
enrolment 
17. Ascertain the students’ characteristics 


that are likely to affect their performance 
as learners and subsequently as health workers. 


18. Gather information about the students 
at the beginning of their training. 


19. Make a summary of the characteristics 
of the student group(s). 


(c) Resources and training processes 


20. Make an inventory of the resources avail- 
able for the programme and describe how they 
are allocated. 


21. Examine and record the training pro- 
cesses used in the programme. 


22. Make a summary of information relating 
to students’ experience, using as themes the 
principal tasks and role models. 


(d) Effects and impact of the programme 


23. Ascertain the effects of the programme 
that are to be evaluated. 


24. Describe and make a critical examination 
of the assessment procedures already in use in 
the programme. 


25. If those methods are not adequate for 
the purposes of the present evaluation, develop 
and apply others. 


26. Work outa set of procedures for observing 
and describing the long-term impact of the 
programme, if that is to be evaluated. 


Phase IV — Analysis and reporting 


27. Prepare a concise description of the pro- 
gramme using the information gathered in the 
course of Phase III. 


28. Analyse the relationship between the 
various aspects of the programme. 


29. Prepare an evaluation report recapitu- 
lating the qualities and achievements of the 
programme, the problems and difficulties 
encountered in carrying it out, and the avail- 
able options or courses of action for its im- 
provement. 
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For further details you are advised to read WHO Offset 
Publication No. 38 Guidelines For Evaluating A Training 
Programme For Health Personnel by F. M. Katz (WHO), 


Geneva, 1978) 


Evaluation of Educational Programmes in Nursing by 
Moira Allen (WHO, Geneva, 1977) 
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points to consider in assessing the 
extent to which programme changes 
foster closer relationships between 
schools for health personnel and the 


wider society 


1. Are changes in the government’s priorities 
concerning health care more clearly under- 
stood by the programme planners inside the 
school? 


2. Does the school now respond more 
swiftly and appropriately to any changes in 
national planning and priorities? 


3. Does the school now work more harmon- 
iously with the ‘‘consumers’’ of the trained 
personnel, i.e. with the relevant government 
agencies and the population? 


4. Is direct feedback from (recent) grad- 
uates’ ‘‘on-site’’ performance used as a criterion 
in assessing the strengths and weaknesses of 
the (new) curriculum? Is the school responsive 
to this type of feedback? 


5. Are the various schools for health per- 
sonnel within the whole national system 
learning from one another and assisting each 
other to replicate successful changes and pro- 
gressively eliminate those which do not seem 
so successful? 


6. Is the course content of the (new) pro- 
gramme really relevant to the common health 


problems of the country? How could it be 


made more relevant? 


7. What observable changes have taken 
place in 


(i) patterns of mortality and morbidity 
or 


(ii) the responsiveness of people to the pre- 
vention of disease and disability? 


Could such changes be the result of the pro- 
gramme changes? 


8. Are consumers more satisfied with the 
quality of health care delivered by the health 
personnel trained according to the (new) 
programme? 


9. What are the principal forces that have 
accelerated the processes of realistic, valuable 
programme change and development? What 
have been the major restraining forces on 
development? How might accelerating forces 
be strengthened and restraining forces altered? 


Based on a paper by P. Blizard, WHO, New Delhi. 
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For each main question try to reply in one of the follow- 


ing ways 


Yes -— 


I have good reason to believe so, and I have 


even obtained some evidence to that effect. 


it. 


Si oow ow Ooo 


I do not think that it is true. 


It is very possible but I have no facts to prove 


I am unable to reply to the question. 


Cy Ee a a eae 


4.07 


points to consider in assessing the 
effectiveness of programme changes 


— in terms of improvement in the administration 
and functioning of schools for health personnel 


1. Has the (new) programme been based on 
a careful study of the skills needed (list of 
tasks) by health personnel in the community 
they are destined to serve? 


2. Have student failure rates and drop-out 
rates been reduced? To what extent? 


3. Has the time from initial enrolment to 
final graduation been reduced? To what 
extent? 


4. Has the volume of content in the (new) 
programme been reduced in comparison to 
what it was previously? To what extent? 


fe eldave there .been reductions in the 
financial and economic cost “‘per graduate’’? 
To what extent? 


6. Are scarce teaching staff and associated 
facilities more effectively and efficiently used? 
In which particular respects? 


7. Is the content of the various subject areas 
vertically and horizontally integrated? What 
are the costs and benefits of that integration? 


8. Are collaboration and cooperation be- 
tween the various departments seen as an 
easier, simpler process than before the pro- 
gramme changes were made? 


9. Have positive (or negative) effects been 
observed in the allocation of resources between 
teaching, research and administration? 


10. Have changes taken place in the teachers’ 
conditions of work so that they can now 
spend more time and energy on their teaching 
and related responsibilities? Have such changes 
been an improvement? If so, why? 


11. Have locally organised teacher education 
programmes had some effect? If so, which? 


12. Has the local education bureau (if one 
exists) contributed to the changes that have 
taken place? In what ways could the function- 
ing of the bureau be improved? 


13. Have administrative tasks become simpler 
and easier or have they become more difficult, 
time-consuming and complex? 


14. Are the roles of the chief administrator 
(Director, Dean, etc.) and other senior admin- 
istrators now clearer, easier and more precise 
or have they become less precise and more 
difficult? 


Based on a paper by P. Blizard, WHO, New Delhi. 
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For each main question try to reply in one of the follow- 


ing ways 


Yes — I have good reason to believe so, and I have 
even obtained some evidence to that effect. 


| ee 


- ee 


Ido not think that it is true. 


It is very possible, but I have no facts to prove 


[am unable to reply to the question. 
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points to consider in assessing the 
effectiveness of programme change 


— in terms of improvements in teachers’ performance 


1. Are the tasks involved in teaching the 
new programme seen by teachers as more 
interesting, relevant and rewarding? 


2. Is the preparation of course content 
seen as a clearer and easier task, notwith- 
standing the possibility that it may be more 
time-consuming? 


3. Have teachers improved their capacity 
to “‘facilitate’’ students’ learning, i.e. have 
they developed the skills necessary to help 
students learn effectively? 


4. Have there been demonstrable improve- 
ments in the programme for training new 
teachers to teach more effectively? 


5. Do teachers now spend more time, effort 
and energy (in productive ways) on their tasks 
as teachers? 


6. Do teachers actually use the skills they 
have acquired during training programmes 
when they are working with students? 


7. Have there been any negative effects on 
teachers’ other roles? 


— and in terms of students’ performance 


1. Are students provided, during their 
studies, with a greater volume of information 
and variety of learning activities? 


2. Do students retain a greater proportion 
of the information and skills they have ac- 
quired? 


3. Do students show an improvement in 
their ability to use these skills in practical 
health care situations? 


4. Do students now show more initiative 
and independence in their learning habits, 
during their time in school, and in the later 
practice of their profession? 


5. Do students show an improvement in 
their willingness and ability to keep up to 
date as regards their professional competence 
(and under adverse conditions)? 


6. Are students now more actively involved 
in the tasks and processes of learning within 
the new programme? 


7. Do studentsnow spend more time, energy 
and effort in their learning activities, and 
generally work harder? 


8. Does the working (and content) of the 
new programme provide a more interesting, 
exciting and rewarding experience for 
students? 


9. Do students play a fuller, more active, 
more rewarding and more useful role in pro- 
gramme planning, implementation and eval- 
uation? 


Based on a paper by P. Blizard, WHO, New Delhi. 
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guidelines for evaluating general 
and intermediate educational objectives 


How can a training school for health personnel 
make sure that the teaching provided meets 
the needs and expectations of the population 
that the future graduates will have to serve? 
One of the best, but under-used, tools to 
ensure such relevance in teaching is the 
educational objective. 


A WHO Study Group met in 1976 to consider 
the place of educational objectives in the 
training of the different categories of health 
personnel, and in particular to devise guide- 
lines for evaluating such objectives. 


The guidelines given below are concerned 
not only with the way in which the objectives 
are stated but also with the process by which 
they were developed. Were the objectives 
defined so as to reflect, for instance, the needs 
of health workers and of the consumers of 
their services? Does the statement of each 


A. Persons involved in drawing 


1. Which of the following persons were 
involved in either developing or reviewing 
the objectives? 


m Teachers (if so please state what dis- 
ciplines or specialties) 


m Practitioners (if so, please state what 
disciplines or specialties) 


= Education specialists 
= Students 


= Administrators (education, health, eco- 
nomy) 


= Consumers of health care (i.e. ‘‘patients’’) 


= Other (please specify) 


2. How was the final decision made con- 
cerning the adoption of the objectives? 


= By me alone (if so, please explain why) 
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objective contain an action verb clearly indi- 
cating what the student must do to show that 
he has attained the objective? Are the proposed 
objectives consistent with one another? 


The guidelines are arranged under 20 headings. 
In most cases, the user can simply check one 
of the answers indicated. A space has been 
left at the end of each item for the user’s 
remarks. 


The report* of the Study Group, and especially 
the proposed guidelines, should be useful to 
administrators and teachers of schools for 
health personnel by helping them to define 
and evaluate educational objectives for their 
own use and for the benefit of their students. 


*Criteria for the evaluation of learning objectives in the 
education of health personnel. Report of a WHO Study 
Group. WHO Technical Report Series No. 608, 1977, 47 pp. 
(The Report is also published in French and Spanish.) 


up the objectives 


= By my administrative supervisor (if so, 
please explain why) 


= By the chief institutional administrator 
(if so, please explain why) 


= By an interdepartmental committee (if 
so, please explain why) 


= Jointly (if so, please indicate who was 
involved) 


= Other (please specify) 


(Circle the answer(s) you wish to give, and 
add any remarks). 


3. Please describe any steps taken to avoid 
the objectives being distorted as a result of 
pressure exerted by certain interested groups 
(specialties, teaching disciplines, or others 
to be specified). 


Remarks: 


B. Basic data and other reference information 


4. Given that the objectives should reflect 
the actual health needs of the community, 
which of the following needs are reflected? 


= Present and future health problems 


x Present and future functions of this 
category of health worker 


= «Functions of other health workers 
= Expectations of health care consumers 
=  &Expectations of health care workers 


=  Interrelationship between health needs 
and other needs of society 


= Official government health policies 


= Structure of the existing and future 
health care system 


m Available data and research on health 
care and health manpower 


= § Current health manpower planning 


5. When the objectives were developed, 
which of the following factors were con- 
sidered? (In each case, please explain how.) 


= Cost to the community (in relation to 
community resources) 


x Characteristics of students 


w Available teaching facilities 


Eo Cultural educational 


institution 


setting of the 


= £xisting educational system(s) and philo- 
sophies 


= Orientation of political and professional 
organisations 


= Other (please specify) 


6. Does each objective describe a specific 
competence that is significantly related to 
the performance of one or more of the tasks 
of that health worker? 


Yes or No 


If ‘‘yes” please give one or more examples. 


Examples: 


7. Do the objectives represent an adequate 
sample of the expected professional compe- 
tences? 


Yes or No 


If ‘ ‘yes’ ? 
derived. 


, please indicate how the sample was 


Remarks: 


Circle the answer(s) you wish to give, and add 
any remarks. 


C. Characteristics of the objectives 


8. Given that the educational objectives are 
by definition student-oriented: 


= Do they relate to the actual work the 
student is going to do after graduation? 


Yes or No 


= Do they describe what the graduate will 
be able to do (using an action verb)? 


Yes or No 


9. Given that the objectives should be con- 
sistent with one another: 


m Do some objectives contradict others? 
Yes or No 


If ‘‘yes’’ please give an example. 


= Do some objectives support others? 
Yes or No 
If ‘‘yes”’ please give an example. 


Remarks: 
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10. Are the objectives realistic with regard 


to: 


© Characteristics of students? 


Yes or No 


cs Characteristics of teachers? 


Yes or No 


i Facilities available? 


Yes or No 


# Time available for learning? 


Yes or No 


11. Given that the objectives should be 
directed towards significant results of learning 
in all domains (i.e. intellectual, practical, and 
communication skills): 


= Are some of them specifically designed 
to facilitate personal affective develop- 
ment? 


Yes or No 


= ~ Are some of them specifically designed to 
facilitate the development of intellectual 
enquiry? 


Yes or No 


= Are some of them specifically designed 
to facilitate development of the ‘‘total 
person’’? 


Yes or No 


= Are some of them specifically designed 
to promote the development of compe- 
tence (and, where appropriate, leadership 
competence) for team work? 


Yes or No 


D. Intended use of the objectives 


12. Has consideration been given to how the 
objectives will be used in your institution? 


Yes or No 


If ‘‘yes’”, please indicate what plans were 
made for their use. 


Remarks: 
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= Do some refer to development of research 
skills? 


Yes or No 
= Does the set of objectives refer to the 
need for life-long self-learning? 


Yes or No 


= Does the set of objectives refer to the 
need for lifelong (continuous) self- 
assessment? 


Yes or No 


= Does the set of objectives reflect the 
scientific base for problem-solving needed 
by that category of health worker? 


Yes or No 


m= Does the set of objectives reflect ade- 
quately the range of technical skills 
required from that category of health 
worker? 


Yes or No 


= Do some objectives reflect the importance 
of the health worker’s responsibility to 
society ? 


Yes or No 


= Do some objectives recognise the need to 
prepare the health worker to be respon- 
sive to new methods? 


Yes or No. 


If ‘‘yes’’, please give examples. 


13. If so, how was the final decision made 
concerning the use of the objectives? 


= By me alone (if so, please explain why) 


= By my administrative supervisor (if so, 
please explain why) 


# By the chief institutional administrator 
(if so, please explain why) 


= By an interdepartmental committee (if 
so, please explain why) 


= Jointly (if so, please indicate who was 
involved) 


= Other (please specify) 


Circle the answer(s) you wish to give, and add 
any remarks. 


14. Are students informed about the objec- 
tives? 


m= Students were given copies of the objec- 
tives. 


Yes or No 


fo] Students have discussed them. 
Yes or No 


If so, under what circumstances? 


x Students have not seen them. 
Yes or No 


If not, why not? 


E. Usefulness of the objectives 


16. As you examine the set of objectives, 
which among the following results do you 
believe likely to be produced? 


= §=© They will be achieved. 
m= They will facilitate student learning. 


= They will facilitate an integrated 


approach to learning. 


= They will facilitate a problem-solving 
approach to learning. 


= =: They will facilitate evaluation. 
= Other (please specify) 
= §=«© They may not even be used. 


Circle the answer(s) you wish to give, and, 
under “‘remarks’’, mention any facts in support 
of your opinion. 


m Some of the students have probably 
seen them. 


Yes or No 
If so, why only some of the students? 
Remarks: 


15. Are teachers informed about the objec- 
tives? 


m All teachers were given copies of the 
objectives. 


Yes or No 
Pe Teachers have discussed them. 
Yes or No 


If so, under what circumstances? 


if Teachers have seen them. 


Yes or No 


If not, why not? 


™ Some of the teachers have probably 
seen them. 


Yes or No 


If so, why only some of the teachers? 


17. Has the existence of the objectives 
had any impact on any of the following 
aspects of the teaching programme? 


=  ‘Teaching/learning activities (if so, please 
give examples) 


m= valuation methods (if so, please give 


examples) 


= Other (please specify) 


Circle the answer(s) you wish to give, and 
add any remarks. 
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F. Evaluation and revision of objectives 


18. Are there provisions for a periodic re- 
view of the objectives? 


Yes or No 

If so, what is the review based on? 
= Students’ performance data. 

= Operational research data. 

m Evidence of changing needs. 

= Other (please specify) 


Circle the answer(s) you wish to give, and add 
any remarks. 


19. If so, who is involved in the review? 
= Teachers 


® Practitioners 


= Education specialists 

= Students 

= Administrators 

= Consumers (i.e. “‘patients’’) 
= Other (please specify) 


Circle the answer(s) you wish to give, and add 
any remarks. 


20. If the answer to question 18 is “‘yes’’, 


= Does such a review facilitate programme 
changes? 


Yes or No 


(If so, please specify how.) 
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The results indicated that the [examination] unduly 
emphasized activities for which there was little patient 
demand and failed to distinguish or require some 
frequently performed procedures. 


Willis, J.B. et al. A preliminary analysis 
of accreditation criteria for surgical 
procedures by orthopedic residents. 

Journal of medical education, 58 
648-654 (1983) 
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evaluation by students of programmes, 
teaching techniques and teachers ' 


& Introduction 


An evaluation made by students can provide 
the teacher with a most useful feedback of 
information on the quality of his teaching. 
Anyone who genuinely wishes to teach better 
should therefore seek his students’ opinions. 
That may be done simply by a friendly talk 
with a few students, but it is preferable to 
prepare and distribute a questionnaire. Such 
an evaluation questionnaire may cover the 
whole or only part of the teaching, and it is 
for the teacher together with the students 
to decide just what should be evaluated by 
them. It would, for instance, be very valuable 
for the teacher to know how the students 
regard any changes in the teaching routine 
such as a new organisation or the intro- 
duction of audiovisual material. 


It may be noted that anyone who asks stu- 
dents to evaluate his teaching need not doubt 
the validity of their judgement. For a decade 
or so many psychometric studies have revealed 
the validity and the accuracy of student 
opinions as well as their close correlation with 
‘objective’ measurements of the instructor’s 
effectiveness. The many biases which were 
ascribed to the evaluators, for example, the 
influence of sex, academic efficiency, (poor 
students/good students), level of studies 
(beginners, finishers) the status of the course 
(compulsory/optional) have all proved to be 
of negligible importance. 


@ Preparation of “‘simple’’ evaluation ques- 
tionnaires 


Once the teaching aspects which are to be 
evaluated have been determined, the eval- 
uation questionnaire is drawn up. 


This questionnaire must comply with several 
essential requirements as concerns the wording 
of the statements, the scale of the answers 
and the method of “administration”. 


1. Wording of the statements 


=  ### The statements should be clear, 
simple and directly related to 
what it is desired to measure; 


= ~=statements which would be accepted 
straight away by everyone should 
not be included; 


= statements containing double ne- 
gatives should be rejected; 


ai each statement should contain a 
complete thought; 


m= the terms “‘uniquely’’, “‘solely’’, or 
‘most of the time’’ should be 
avoided or used in moderation; 


= a statement should preferably be in 
the form of an affirmation concer- 
ning which the student is asked to 
express a favourable, neutral or un- 
favourable opinion; 


= the number of statements should be 
limited. Too long a questionnaire 
wearies the evaluators and this ine- 
vitably affects the validity of the 
answers. It is estimated that a 
questionnaire should not exceed 
60 statements. 


; Summary of a paper presented by J.-F. d’Ivernois, 21 May, 
1975, Séminaire de Pedagogie universitaire, Laval University, 
Quebec. 
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1. Examples of statements 


“Specific objectives make it possible to 
identify what is most important in the content 


to be learned’’. 


“The time allocated for a learning activity is 
sufficient for you to achieve all the objectives 


aimed at by that activity”’. 


2. Scale of answers 


The student should not have to answer merely 
“yes” or “no’’ to a statement, for such an 
answer provides only scanty or dubious infor- 
mation. 


That is why preference is given to the use of 
scales of answers comprising 5 degrees (and 
sometimes 7 or 9) (see below). 


Students should enter against each statement 
(or on an answer-form or card) the numbers 
corresponding to their opinion. It is advisable 
to provide for an additional answer corres- 
ponding to a lack of opinion concerning the 
statement (coded 0). 


The answers scale should be explained to the 
students at the beginning of the questionnaire, 
or mentioned against each statement. It is also 
as well to leave a space below each statement 
for comments by the student. 


3. Method of administration 


A person’s reaction to a given stimulus is first 
immediate and then delayed. If the question- 
naire is handed out immediately after the 
event the results observed will be different 
from those obtained if the questionnaire is 
administered later. These differences in dis- 
tribution are attributable to the effect of 
experience (also called the ‘‘practice effect’’). 
The teacher should therefore take this factor 
into account when he envisages the adminis- 
tration of an evaluation questionnaire. 


4. Analysis and interpretation of the answers 


Processing (machine or manual) of the answers 
is simple. The answers to a given statement are 
then analysed by calculating the number of 
students (frequency) who have replied by 
1, 2, 3, 4, 5 or 0 to that statement. The mean 
of the answers to the statement can also be 
calculated. To do this one point is assigned to 
an answer “1”; two points to an answer “2”; 
three points to an answer “3”; etc. The total 
of the number of points obtained for the 
statement is calculated and then divided by 
the number of students answering. To make 
interpretation easier, the trends (favourable, 
unfavourable) should be grouped together. 


A scale ranges from disagreement (1 and 2) through neutral* (3) to agreement (4 and 5) 


Example: 


«—__—___— Disagreement —__—____» <_______——. Agreement ———_» 


Noopinion = Complete Partial 


Neutral Partial Complete 


*For certai i i : . : 
ee a he eecanre it may be considered that a participant in an educational workshop, for instance, should be obliged to 
P position and not take refuge.in neutrality: the same applies to the position “no opinion’’. 
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Example 


Statement: “The time allocated for a learning 
activity is sufficient for you to achieve all the 
objectives aimed at by that activity”’. 


Numbers of answers: 100 


Distribution: 1 : 3 $535 5 : 69 


Trends: 1 — 2 (disagreement) 
4—5 (agreement) : 90 


Mean of answers : 4.51 


© Construction of complex questionnaires 


1. “Simple” questionnaires enable the stu- 
dents’ perception of reality to be measured. 
It is for the teacher who analyses the answers 
to deduce from them what the students desire 
or expect, but this interpretation may leave 
certain grey zones. If, for example, the state- 
ment submitted is: ‘the course provides an 
adequate coverage of the subject matter” and 
40% of the students disagree with that state- 
ment then it is clear that these students feel 
that the course does not cover sufficient 


subject matter. On the other hand, what pro- 
portion of the 50% of students who agree with 
the statement think that the subject matter 
covered is not only adequate but even too 
much? 


“Complex” evaluation questionnaires, such as 
the one devised by F. Gagné, covering the 
perception by the students of the teacher/ 
students relationship have the aim of mea- 
suring as unambiguously as possible both the 
students’ perception of reality and their level 
of expectations. 


In a questionnaire of the Gagne type, each 
statement should include: 


= § atitle; 


= a detailed description of the aspect to be 
measured ; 


m A 5-degree scale with qualitative ex- 
pression of each degree; 


= = Two questions: the first (a) measuring 
the student’s perception of reality and 
the second (b) the level of his expect- 
ations. 


Frequency of problem-solving activities: What is required is to measure the frequency of the 
learning activities requiring students to search for solutions to priority health problems. 


Very rare examples 

Rare examples 

More or less frequent examples 
Frequent examples 


Very frequent examples 


Question A: Where would you place this course on the evaluation scale? 


Question B: Where should it be in order to satisfy your 
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2. Analysis and interpretation 


The means of the answers to questions A 
(Reality) and B (Expectations) are calculated 
for each statement. Interpretation of these two 
means is easy since degree 1 on the scale shown 
usually corresponds to the minimum frequency 
or intensity, while degree 5 corresponds to 
maximum frequency or intensity (see 
example). The deviation between these two 
means is then calculated (S score). This de- 
viation shows the dissatisfaction of the stu- 
dents, a dissatisfaction expressed in terms of 
a lack or an excess. The lower the value of 
the S score (the nearer it is to zero) the more 


the particular aspect measured is deemed 
satisfactory. The S scores of several state- 
ments can be compared with one another 
for one and the same group of students. 
Furthermore, all the S scores in the question- 
naire can be added together so as to give an 
overall and valid measurement of satisfaction 
or dissatisfaction. Finally, it should be noted 
that a questionnaire of the Gagné type can be 
adapted to different educational methods, 
e.g. lecture courses, small group activities, 
teaching by computer, by television, etc. 


Example of Interpretation of a Questionnaire of the Gagné type. 


Statement Title of statement 


Variation in 
educational approach 


Active student 
participation (+ or — ) 


Number of references 
to be consulted. 


Mean : 
Reality 


Deviation 
(S score) 


Mean : 
Desires 


Interpretation 


3.74 ; Dissatisfaction 


4.65 : Satisfaction 


Dissatisfaction 
(excess) 


Gl Ge i. a) 0) OO. a a a a ae ee 


On this subject, see also Evaluation of teachers and 
teaching effectiveness by Christine H. McGuire in WHO 
Public Health Paper No. 61, Geneva, 1974. 


Yop Uo oD OO OOD oD OO hain Seen 


4.18 


List the advantages and limitations of this type of evaluation of the educational 
students. 


Advantages | Limitations 


process by 
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evaluation of students’ level 
of performance 


The following pages contain some examples of tests. In each case you are advised to practise 
making up a test on a subject with which you are familiar, after having selected a specific 
educational objective whose achievement can be effectively measured by the test in question 
(i.e., the test must be valid). 


pages 
Real or simulated practical test 4.22 
Execution of a project 4.22 
Observational rating scale 4.23 
Oral test - 4.27 
Essay 4.28 
Short, open answer questions 
(modified essay question, and restricted 
response test) 4.28 
Multiple choice questions (MCQ) 4.31 
Programmed examination 4.41 


Excellent examples may be found in 
WHO Public Health Paper No. 72 ‘Assessing Health Workers’ Performance: A Manual for 
Training and Supervision’, Geneva, 1980 


and 


Catalogue of instruments for the performance assessment of Health Workers , HMD/81.6, WHO, 
Geneva, 1981. 
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assessment of professional skills 


Real or simulated practical tests 


These tests are based on the direct observation 
of a professional task (cf. page 2 22). 


Ld 


A practical test is one that requires the student 
to perform a professional task in an environ- 
ment and under conditions the same as or 
similar to those in which he will have to per- 
form it in his future professional life. 


For example: 


The student is required to weigh a baby in an 
MCH centre and note the result on the appro- 
priate record card. 


The student measures the blood pressure of 
another student. | 


The student is required to prepare a blood 
slide. 


It is advisable to use this technique when the 
main component of an educational objective 
is a practical skill or an interpersonal relation- 


ship. 


The disadvantages of this type of test are: the 
relatively high cost in teaching personnel in 
view of the time required to observe each 
student’s work; the risk of the patient being 
placed in a disagreeable situation ; the difficulty 
in standardising the test conditions; and the 
heavy administrative task of coordinating the 
time schedules of teachers, students and 
services. 


Execution of a project 


These tests are based on the indirect obser- 
vation of a professional task. 


A project execution test is one that requires 
the student to carry out an activity, in a 
variable period of time, that results in a 


product which is to be evaluated by the 


teacher. 


For example, the product might be a concrete 
piece of work such as a dissection, a dental 
impression or a_ histopathological slide; in 
addition to practical skills, certain intellectual 
processes can be evaluated by projects in a 
written form, such as a research report or a 
bibliography. 


It is advisable to use this technique when 
the main component of an _ educational 
objective is a complex practical or intellectual 
skill, and when the product is more important 
than the student’s manner of working. 


The disadvantages of this type of test are: 
the relatively high cost in teaching personnel 
responsible for evaluating the result of the 
project; and the need to establish a relation- 
ship of confidence with the student to avoid 
cheating. 


Bea OO OO ONO 0 Oo ee Ooo 6 ole Boo Gee 


Draw up a practical test, either real or simulated, or else a project test, designed to assess a 
professional task (pp. 1.41 or 1.52 or any task chosen by you), and keeping in view the criteria 
of validity, objectivity and practicability (re-read p. 2.38). 


Evaluate your results by constructing a specification table (see pb. 3.73}. 


Si eo we ei eo ep Bogs Cefn fief ee eho Peale a Oe 
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assessing attitudes by 
observational rating scale 


Everybody agrees that the attitudes of the 
physician and the other members of the 
health team to the patient and the patient’s 
family are of the greatest importance. 


Yet when one observes the way in which stu- 
dents’ skills tn this matter are assessed, it is 
seen to be so inadequate that one cannot help 
being struck by the paradox of the situaton. 


It must be recognised, in all humility, that 
this is the most difficult domain in which to 
make assessments, and the efforts of research 
workers in this field deserve every support. In 
a book such as the present one, only a super- 
ficial treatment of this subject can be given, 
and the reader should refer to more specialised 
publications (see bibliography). 


1. Selection of students and assessment of 
attitudes 


The term “‘attitude”’ denotes certain constant 
traits in an individual’s ways of feeling and of 
thinking, and his predispositions towards 
action with regard to another person such as a 
patient or collaborator. An attitude is generally 
considered to be a hypothetical construct 
which is not directly observable but can be 
inferred from speech or outward behaviour. 
The inventory of attitudes constitutes the 
operational definition of the attitude. 


Attitudes are probably not innate. The whole 
personality structure of an individual, and 
thus the whole of his behaviour, is constituted 
by a complex of interlinked attitudes. In its 
present state, research seems to indicate that 
it is illusory to expect to be able to change 
attitudes in the relatively short period of a 
programme of higher studies. Perhaps then it 
would be preferable to make sure, by a suitable 
selection process, that students at the begin- 
ning of their studies already possessed the 
appropriate attitudes for their future pro- 
fession. It must however be remembered that 
the definition of those attitudes may easily 
be contestable; the preparation of such de- 
finitions must therefore be a group activity 


free from arbitrary influences. It would be 
wise, moreover, to conduct carefully con- 
trolled experiments before taking any action 
based on the results of attitude assessment. 


The situation being what it is, the drawing of 
lots may not unreasonably be considered as 
a method of selection.* 


2. Observational rating scales during studies 


The method of observational rating scales re- 
quires the repeated and standardised direct 
observation of students’ activity over a long 
period (several months) and in natural pro- 
fessional situations such as an outpatients 
clinic, a consulting room, a laboratory, or a 
hospital ward. The rating scales are used in 
evaluating the students’ reactions, attitudes 
and activities. The fact that the scales are 
very easy to use often leads people to forget 
the many. possible sources of error in such 
evaluations. 


If a student is being observed when examining 
a patient, and if it is wished to make a separate 
evaluation of how he gains the patient’s con- 
fidence, the following rating scale may be 
used. 


The student has taken all the necessary pre- 
cautions, and the patient appears completely 
relaxed ia) h38 oe ee 5 


The student has taken the necessary precau- 
tions and has reassured the patient several 
TLIMES 2%. sesso Ss ten ae ee ss 


The student has made an effort, and has 
followed, Iti pi. 4, egestas oer eee 3 


The student has made an effort, without 
folloWing YUP. BOR set a era 2 


The student seems to be quite unaware of the 
PEOGIEMe% o.. tects iO er aww mes oes wee 1 


*The selection of students is made by drawing lots in the 
Netherlands. 
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an example of an attitude rating scale 


Task/Educational Objectiv 
hospitalised child 


Attitudes 


1. When giving the 
mother infor -. 
mation on her 
child’s condition 


2. When giving a 
clear explanation 
of what has been 
done for the 
child 


3. Whensuggest- 
ing that the 
mother should 
see her child 


4. When telling 
the mother how 
she can obtain 
information 
about her child’s 
condition 


5. When suggest- 
ing to the mother 
how a rupture of 
the mother/child 
relationship can 
be avoided 


refuses to reply 
to the mother’s 
questions 


often uses 
medical terms 
without ever 
explaining 
their meaning 


refuses the 
mother’s 
request to see 
her child 


tells her 
nothing 


refuses to say 
how when the 
mother 
enquires 


gives no infor- 
mation 
spontaneously 


often uses 
medical terms 
and seldom 
explains their 
meaning 


does not 
suggest that 
the mother 
should see her 
child 


when the 
mother 
enquires, 
refers her to 
the nurse 


makes no 
suggestion 


gives inaccurate 
information 


seldom uses 
medical terms 
but does not 
always explain 
their meaning 


agrees when the 
mother makes 
the request 


when the 
mother 
enquires, tells 
her the visiting 
hours and the 
persons she 
should ask 


puts off the 
question until 
later 


gives accurate 
information but 
does not reply 
to mother’s 
questions 


seldom uses 
medical terms 
and always 
explains their 
meaning 


spontaneously 
suggests that 
the mother 
should see her 
child 


spontaneously 
informs the 
mother how 
to obtain 
information 
about the 
child’s 
condition out- 
side official 
hours 


spontaneously 
suggests that 
the mother 
should stay at 
the hospital 
with her child 


Criterion: The student should score 7 out of 10 on the above rating scale 


After Professor J. -P. Grangaud (Test and measurement workshop, Timimoun, 1977) 


e: During a telephone conversation, to reassure the mother of a newly 


gives accurate 
information and 
replies to mother’s 
questions 


uses only terms 
that the mother 
can understand 


spontaneously 
makes the 
suggestion and 
explains any 
precautions to 
be taken 


encourages the 
mother to come 
and obtain 
information 
about her child’s 
condition outside 
official hours 


attempts to over- 
come the 
difficulties in the 
way of the 
mother staying 
with her child 


The above example uses one of the most frequently employed rating scales. Others may be found 
in The assessment of attitudes by Agnes G. Rezler, WHO Public Health Paper No. 52, Geneva 
1973, and in J. P. Guilford (1954) Psychometric methods, McGraw Hill. | | 
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Select a professional task (educational objective) whose main component is an attitude and con- 
struct a descriptive rating scale for its evaluation. 


Task/Objective: The student should be able to: 


Attitudes 


Criterion/acceptable level of performance: —— $$$ 
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Try once more — it is not easy. Show your first attempt to several of your colleagues separately; 
discuss it in a small group including users of health services 


Task/Objective: The student should be able to: 
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a few words about the traditional! 


oral examination 


Definition: An examination consisting of a 
dialogue with the examiner who asks questions 
to which the candidate must reply. 


In its standard form, the oral examination is 
a closed-book test. In that form it can evaluate 
only level 1 educational objectives (see page 
1.38) in the domain of intellectual skills. Like 
traditional written examinations using short, 
open answer questions or MCQ,, it provides a 
check on whether the student can express, 
more or less clearly, his knowledge of isolated 
facts or groups of facts that he ought to 
remember. Most often, it takes the form of a 
series of not necessarily interrelated questions. 


There is a dialogue only if the examiner so 
wishes. 


It should be pointed out that, apart from its 
advantages and limitations which were des- 
cribed on page 2.30, this type of examination 
suffers from a scarcity of examiners who are 
really capable of making the best use of it in 
practice, 


A better way of assessing a student’s ability 
to communicate orally with another person is 
to use simulation methods, such as role playing 
or a telephone conversation, which are much 
nearer to actual professional tasks. 


Personal Notes 
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long and short written questions 


The use of the “essay question” 
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Use “essay questions’ only to evaluate a 
type of performance which cannot be 
measured just as efficiently by other 
methods (synthesis of a group of com- 
plex concepts, summary of a document, 
comparjson of two phenomena, cause 
analysis, finding relationships, criticising 
the relevance of a concept, formulating a 
plan of action). 


Limit the problem posed so that it is 
clearly apparent to the candidate and 
define the structure of the answer. 


Employ terms that are as explicit as 
possible, such as ‘‘summarise’’, or 
econipate., evaluate. detine, 4* “ar 
range in order’’, etc. rather than “‘discuss’’ 
or ‘‘state everything you know’’, so that 
all the candidates immediately know 


what they have to do. 


Choose problems which call for careful 
consideration but whose solution can be 
briefly set out in the time allowed. 


Allow no choice among the questions 
SCL 


For every question, set out yourself the 
elements which, according to you, should 
appear in the answer (scoring procedure). 


Mark papers anonymously. 


When two or more teachers correct the 
same test, they should agree on the 
scoring procedure before the test and 
correct the results separately. 


Use a point system of scoring based upon 
those elements that are expected to 
appear in the answers. 


Try out the scoring procedure on a few 
papers. Preferably, have all the teachers 
then read all the answers to a given 
question; or, if need be, have one of 
them read all the answers to a particular 
question in all the papers, have another 


teacher do the same for another question, 
and so on. 


Score the answers of all the students to 
one question before going on to the 


scoring of another question. 


= Do not form a judgment of a candidate 
on the basis of only one question but 
calculate for each candidate a summative 
score based on the reading of several 
different essays, since such scores are 
more reliable than the score for any one 
essay taken separately. 


Use of short, open answer questions 


This involves series of questions drafted in such 
a way that the answer calls for a predetermined 
and precise concept. As their name indicates, 
the answer expected is short and can be 
expressed in different forms (open). Ideally, 
only one answer is acceptable no matter in 
what terms it is expressed. 


The author of the question must define in 
advance (and in cooperation with colleagues) 
the answer called for by the wording of the 
question. If it appears that conceptually 
different answers will do for a given question 
then it should be reworded until that draw- 
back disappears. 


The pagination should allow the necessary 
space for the answer below each question. 
Marking is theoretically simple since the 
answer has been predetermined. In practice 
the frequency of difficulties arising from the 
conflicting views of two correctors is found to 
be of the order of 2%. There remain the pro- 
blems of illegible writing,* absence of answers, 
copying mistakes, etc. 


Moreover, all the rules concerning essays apply. 


Short, open answer questions are also called 
“restricted response tests’’.* * 


A set of short, open answer questions preceded 
by a case history is sometimes called: Modified 
Essay Questions.*** 


*The same problem arises in the use of MCQ with automatic 
marking, but its frequency does not exceed 0.4%, and is very 
often less. 


**Rezler, A. G. The restricted response test: one alternative 
to multiple choice and essay tests, in NLN Publ. (15 — 1761), 
National League for Nursing, New York, 1979. 


***Knox, J. D. E. The modified essay question, Dundee 
Association for the Study of Medical Education, 1976 
(Medical Education Booklet No. 5). 


Following some of the principles set out on the preceding page, prepare a written, ‘‘open-book’’* 
examination question of the essay type, and three short, Open answer questions; indicate the 
standard of performance for each so that marking can be objective, i.e., a rating scale comprising 
all the information that another person will need to be able to mark in the same way as you. 


Written question of the essay type, including the text of the expected answer. 


Scoring instructions: Acceptable level of performance 


*An “open-book”’ question means that the candidate may refer to any book, document, handout or personal notes. Such a 


question is not, therefore, intended to evaluate ability to memorise. 
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Now go on to draft some short, open answer questions. 


Text of the question 


Scoring Instructions: Acceptable level of performance 
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Acceptable answers 


directions for writing 
multiple- choice questions 


1. Make certain that the stem consists of a 
complete statement, not just a single word. 


2. Place all common elements in the stem 
of the item. This adds simplicity and com- 
pactness to the item. 


3. Make each item completely independent 
of answers to other items (for instance, the 
stem of one should not suggest the answer to 
another). 


4. Eliminate all unrelated details from an 
item. 


5. In general, avoid negative statements, but 
if a negative expression does appear in the 
stem of the question, underline it to draw the 
student’s attention to it. 


6. Use plausible or logical distractors. Each 
distractor should, by its content or nature, be 
such that it appears to have something to do 
with the question. Unrelated distractors appear 
silly to a thoughtful examinee. Since the 


number of possible answers is thereby cut 
down, the item loses some of its value. 


7. Avoid the use of clues that may suggest 
the correct answer. 


8. Be sure that the distractors and the 
correct response possess homogeneity, that 
is, they should be fairly similar in content or 
in the total number of words. 


9. Be cautious of the use of ‘‘none of the 
above’”’ as a distractor or aS a correct answer. 


10. If it is impossible to obtain more than 
three plausible responses, do not waste time 
trying to invent some others. 


11. When dealing with items that have 
numerical answers, arrange the answers in 
order from large to small or vice-versa. 


12. Arrange the place for the correct answer 
in such a way that, for the test as a whole, no 
letter corresponding to a given answer appears 
more frequently than some other letter. 


examples of multiple-choice questions 


One ‘‘best response’ type 


Question 1 

In differentiating cirrhosis of the liver from 
chronic constrictive pericarditis, a useful 
physical sign is: 

(a) hepatomegaly ; 

(b) ascites; 

(c) distension of the neck veins; 

(d) pitting oedema of the ankles and legs; 


(ec) splenomegaly. 


Question 2 


Active immunisation is available against all of 
the following diseases except: 


(a) tuberculosis; 
(b) smallpox; 

(c) poliomyelitis; 
(d) malaria 


(e) yellow fever. 


The multiple-true-false type (also called ‘‘mul- 
tiple-response item’’). 


This type consists of a stem followed by several 
true or false statements. The candidate is to 
determine whether or not each of the four 
statements which follows is true or false. He 
then responds according to a code which 
permits one out of five possible combinations 
or responses whereby one, two, three, or all 
four statements may be true. 


— when properly written, the multiple-true- 
false item type tests the student’s know- 
ledge or understanding of several related 
aspects of a substance, a disease, or a 
process ; 


— each of the statements or completions 
offered as possibilities must be clearly 
true or false. This is in contrast to the 
type “1” format in which alternatives 
which are “partially correct” may be 
used as distractors; 


— this type of item should be written so that 
no two of the alternatives are mutually 
exclusively, i.e. the answer “‘all are 
correct”’ must be a possible response. 
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The directions for this item type are as follows: 


For each of the incomplete statements below, 
one or more of the completions is correct. On 
the answer sheet blacken space under: 


(a) if only 1, 2 and 3 are correct; 
(b) if only 1 and 3 are correct; 
(c) if only 2 and 4 are correct; 
(d) if only 4 is correct; 


(e) if all are correct. 


Question 3 


A child suffering from an acute exacerbation 
of rheumatic fever usually has: 


1. an elevated sedimentation rate; 

2. aprolonged P - R interval; 

3. an elevated antistreptolysin 0 titre; 
4 


subcutaneous nodules. 


The matching type 
Directions for constructing matching items’ 


— Limit the number of entries to about 10. 
If situations arise where 20 or 30 entries 
must be considered, construct two or 
three matching items. When long lists 
have to be matched, the student wastes 
too much time in trying to find the 
correct response. 


— Do not break items by the bottom of the 
page. The complete item should be on the 
same page. 


— Have a longer list of questions than of 
possible answers and state in the direc- 
tions that these may be used more than 
once. When there are an equal number of 
questions and answers, it is possible for 
the student, after responding to some of 
them, to complete his task by elimination 
and guessing. 


— Strive for homogeneity. 


: Downie, N. M. Fundamentals of Measurement: Techniques 
and Practices, New York, Oxford University Press, 1967. 


The directions given to examinees for this type 
of item are as follows: 


“Each group of questions below consists of 
lettered headings followed by a list of num- 
bered words or statements. For each numbered 
word or statement, select the one heading that 
is most closely associated with it and blacken 
the corresponding space on the answer sheet. 
Each lettered heading may be selected once, 
more than once, or not at all. 


Examples: 


Questions 4 to 9 
(a) increased metabolic activity; 
(b) hyperinsulinism; 
(c) lack of storage of glycogen in the liver; 
(d) storage of an abnormal glycogen in the 
liver ; 
(e) decreased secretion by pituitary or 
adrenal glands; 
adenoma of islets of Langerhans; 
violent physical exercise; 
hyperthyroidism; 


4 
5 
6 
7. Simond’s disease; 
8. Von Gierke’s disease; 
9 


epidemic hepatitis; 


Questions 10 to 14 

(a) sodium bicarbonate; 

(b) sodium carboxymethylcellulose ; 

(c) aluminium hydroxide gel; 

(d) none of the above; 

10. a gastric antacid which is also used 

in the therapy of hypoparathy- 
roidism because of its property of 


reducing the absorption of phos- 
phorus; 

11. because it is absorbed, it may cause 
alkalosis, particularly in infants and 
elderly patients; 

12. a gastric antacid which has the disad- 
vantage of causing ‘‘acid-rebound”’; 

13. a gastric antacid which precipitates 
and inactivates gastric pepsin ; 

14. a gastric antacid and demulcent 
which can be converted to liver gly- 
cogen. 


The comparison type 


The ‘‘comparison”’ type permits one to com- 
pare and contrast two diseases, signs, symp- 
toms, laboratory findings, etc. 


When using this type of item, one must be 
careful to: 


= ~= avoid the trivial; 


m avoid selecting as one of the pair some- 

thing that is rare or unusual. For example, 

if the item asks about the relation of a 

certain symptom to disease ‘‘x”’ or “y”’, 
and the frequency of the symptom in 
the two diseases is 90% and less than 1% 
respectively, then the examinee is in a 
dilemma. If he follows the principle of 
the ‘‘general rule”, he may select answer 
A (‘‘x” only); but if he is aware that the 
symptom does occur in the exceptional 
case of disease “‘y’’, then he may select 
answer C (‘“‘both’’). Which response is 
correct? 


The instructions for this type of item are as 
follows: 


“Each set of lettered headings below is fol- 
lowed by a list of numbered words or phrases. 
For each numbered word or phrase, blacken 
the space on the answer sheet under: 

(a) if the item is associated with (a) only; 
(b) if the item is associated with (b) only; 


(c) if the item is associated with both (a) 
and (b); 


(d) if the item is associated with neither (a) 
nor (b). 


Examples: 


Questions 15 to 17 
(a) hookworm disease; 
(b) ascariasis; 
(c) both; 
(d) neither; 
ee eosinophilia; 
16. hypochromic anaemia; 


17. infection through the skin. 


Questions with diagrams, photographs, etc. 


Questions 18 to 20 


Level of : 
+- —— — --- 18. Total lung capacity 


19. Inspiratory capacity 


maximal expiration 


20. Vital capacity 


eo Ct 4 oO OD. Geo See ese Possess 


Test and measurement, or the study of tests used in 
measurement techniques, is a fairly new science. It was 
introduced into the world of health sciences teaching 
about 50 years ago against some opposition, and the 
problem has certainly aroused the interest of teachers; 
however, some of them feel that the evaluation 
‘‘specialists’’ are trying to poach on their preserves, and 
that this will limit their academic freedom. This is often 
due to a lack of information. 


Better information would help to disarm the defence 
mechanisms displayed by teachers when the problem is 
tackled scientifically, and would thus reduce heated 
reactions. 

BE oh. GOs ao Don OOo eee 


de oOo OO 0 8.0 0.0 Oo O Dele ie Grae 


If no reference is made to questions of the true-false 
type it is not a chance omission! They are really very 
bad and should not be used. 


Bet) GO Ge oo Oo Bn Oo 2 eee 
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examples of items constructed in 
accordance with the level of 
intellectual processes* 


Utilisation of clinical tests for application of 
basic science concepts. 


Level 1 — Recall of facts 


Item 1: 


The Fick principle is illustrated mathematically 
by which of the following formulae: 


Ao, ~ Vo, 


ie O, Consumption (ml/min) 


Cardiac Output (litres/min) 
Heart Rate (beats/min) 


Heart Rate (beats/min) 
Cardiac Output (litres/min) 


O, Consumption (ml/min) 
Agate V5 


2. 2 


Cardiac Output (litres/min) 


Body Surface Area (sq. metres) 


Level 2 — Interpretation of specific data 


Item 2: 


Apply the Fick principle to the following data 
and calculate the amount of glucose (in mg/ 
min) extracted by the spleen of a dog: 


Cardiac output — 2 litres/min 
Stroke volume — 200 ml/min 
Cardiac index — 2.8 litres/sq. metre 
Arterial glucose — 100 mg/ml 
Splenic vein glucose — 96 mg/ml 
Splenic blood flow — 50 ml/min 


*Re-read pages 137-140 


Which of the following corresponds to the 
result of your calculations? 


A 140 
Bo 78 
C 200 
Ly 22) 
E 250 


Clinical case (for questions 3, 4 and 5) 


A 25 year old male patient comes to your 
office complaining of cramps in the muscles 
of his arms or legs whenever he engages in 
strenuous muscular exercise. This complaint 
has become more prominent over the past 
three to four years. The general physical 
examination is normal. The patient has 
previously been seen by another doctor who 
performed a muscle biopsy. When you call the 
other doctor’s office, you learn that the only 
abnormality noted on the biopsy was excess 
accumulation of glycogen in the muscle fibres. 
Electromyography was normal. The blood 
chemistry determinations related to carbo- 
hydrate metabolism which are performed in 
the hospital where you work are glucose and 
lactate. 


You decide that you want to find out why 
the patient has muscle cramps and excess 
glycogen. To obtain further data, you apply 
the Fick principle to the right leg and insert 
small polyethylene catheters in the femoral 
artery and femoral vein at the level of the 
inguinal ligament. You make a blood flow 
estimation for the right leg and obtain a 
value of 200 ml/min at rest. You also take 
two resting blood samples from the arterial 
and venous catheters. 


You then have your patient pump a bicycle 
ergometer for five minutes and at the end of 
exercise you take a series of five arterial and 
venous samples at two minute intervals. You 
have glucose and lactate determinations 
performed on each of the 14 blood samples. 


The results on the resting samples are reported 


as follows: 


Item 3: 


The glucose and lactate extraction (in mg/min) 
by the tissues of the right leg in the resting 
state Is: 


ae e-aty eaee e 


Level 3 — Solution of a problem 
Item 4: 


On the basis of your knowledge of muscle 
carbohydrate metabolism, the history from 
the patient and the muscle biopsy report, 
which of the following sets of pre-post 
exercise data on venous blood would be 
most likely in this patient? 


200 mg% 20 mg % 


Rest ;Ex. est Rest | Ex.! Rest 
Min. Min. 


Glucose Lactate 
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200 mg % 20 mg % 
B 
Rest | Ex. | Rest Rest Ex.| Rest 
Min. Min. 
Glucose Lactate 
200 mg % 20 mg % 
C Rest Ex.] Res | Rest | Ex. [Best 
Min. Min. 
Glucose Lactate 
200 mg % 20 mg % 
D 
Rest [Ex.]| Rest Rest Ex. {Rest 
Min Min 
Glucose Lactate 
Item 5: 


Study the basic principles of carbohydrate 
metabolism as shown in the figure on page 
4.37. Note those parts of the metabolic route 
indicated by a letter (A, B, C, D and E). 


Assuming that one of several reasonable 
explanations of the symptoms presented 
by the patient is a metabolic block, and on 
the basis of the data available to you after 
study of the preceding parts of the question, 
select the letter which would be the most 
probable site in this route of metabolic 
block in the patient. 


Mark on the answer sheet the letter identi- 
fying the point in the route where a more 
thorough study would have most chance of 
giving positive results. 


Gly cogen 


Uridine 
Di-Phospho- 


Carbohydrates Glucose 
Phosphorylase 
ATP-ADP 
Glucose ———————_—_»> 


sabe pee Glucose Glucose-1-PO4 


Glucose-6- 


Cell Wall Phosphates 


Glucose-6-PO4 ities toe 
< unt 


. Fructose-6-PO4 


ATP-ADP 


Phosphofructokinase Fructose-1,6-Diphosphates 


Fructose-1,6-Diphosphate 


(2 mols per mol of glucose) =§ —————— > Glycerol 
Phosphoglyceraldehyde | “<—————- (from fats) 


ADP eee ATP 


DPN ———» DPNH 
Phosphoglyceric acid 


Enolphosphopyruvic acid 


Proteins “———~> Alanine @-———>_ Pyruvic acid Lactic acid 
Acetyl-CO A 


Aspartic ————————-_ Oxaloacetic oe 


acid <—__________——_ acid Fats 


Citric acid 


Citric acid 
cycle 
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Any examination question that has not been tried out 
on a representative student sample should not be used 
in a certifying test. 
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Compose one or two multiple choice questions of each of the types described above (one best 
response, multiple true-false, matching, and comparison types). At least half of your questions 
should measure an intellectual process above level 1, recall of facts (either level 2, interpre- 


tation of data, or level 3, problem-solving). Use the objectives that you yourself drew up on 
pages 1.54 and 1.55. 
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You should now be able to reply ‘‘yes’’ to each of the following questions about the items you 
have just composed. Refer also to the directions given on page 4.31. 


In general 


1. Is the item realistic and practical? 

Does it deal with a matter that is professionally useful and important? 
ls it drawn up using the technical language of the profession? 

Does it require intellectual skills of a professional kind? 

Is it independent of every other item in the test? 


Is it specific? 


Lia ee veer gee ts te 


Does it avoid the error of giving away the correct answer by irrelevant details 
or extraneous data? 


The essential problem 


8. Is it clear? 
9. Is it stated in precise terms? 
10. Is it stated briefly and completely? 


11. Does it contain only data related to the answer? 


The distractors 


12. Are they important, p/ausible answers rather than obvious distractors? 


13. Do they deal with similar ideas, or data expressed in similar form? 


An examination must have regard to practicability 


Whether it is practicable will depend on the time nec- 
essary for its construction and administration, and the 
scoring and interpretation of results, as well as on the 


general ease of its use. Examination methods that lack 
practicability become a heavy burden on the teacher, 
who will then tend to give less than due importance to 
the measuring instrument. 


Simulation has the advantage of coming nearer to reality 


while permitting standardisation and protecting the 
patient. 
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the programmed examination 


The advantages of this relatively recent method 
(it appeared at the beginning of the 1960s) are 
so great that they should help to compensate 
for the difficulties attaching to its use. 


Briefly, its aim is to measure (by simulation 
on paper) the problem-solving component of 
clinical competence. 


Like the method of multiple choice questions, 
it is highly objective and can be corrected by 
computer. Its name shows that this new exam- 
ination method has certain aspects identical 
to those of programmed teaching, where 
the candidate advances, step by step, through 
a series of consecutive clinical problems. 


The method was developed in the United 
States of America and several types of such 
simulation tests can be found in the literature. 
They are sometimes referred to as Patient 
Management Problems (PMP), Clinical Sim- 
ulations, etc. 


Objectives of the method 


The aim of the method is to evaluate clinical 
competence: 


— Ability to detect and satisfactoirly inter- 
pret abnormal signs and symptoms. 


— Ability then to reach a reasonable diag- 
nosis and to show satisfactory judge- 
ment in the choice of treatment. 


Until 1963, examiners tried to find an answer 
to these questions by confronting the candi- 
date with a carefully selected patient. This 
method was effective in the past, when 
candidates were not very numerous. More 
recently, faced with thousands of candidates, 
thousands of patients and thousands of 
examiners, test specialists confronted a 
difficulty which they rapidly recognised. 
There were three variables: the candidate, 
the patient and the examiner. This represented 
two variables too many for a valid evaluation 
of the candidate. 


The first research aim was to seek a valid 
definition of the qualities involved in what is 
termed clinical competence (at the level, for 
example, of a hospital intern). One method 
employed was that of the questionnaire using 


Flanagan’s ‘‘critical incident”’ technique (see 
Pp. 1.09). 


Through direct interviews and questionnaires, 
some 600 physicians were asked to describe 
clinical situations during which they had 
personally observed interns in the course 
of their work and had been impressed on 
the one hand, by examples of satisfactory 
clinical conduct, and, on the other by ex- 
amples of unorthodox clinical conduct. Three 
thousand situations of this type were analysed. 
This ample documentation gave an idea of 
what had to be evaluated. 


The following step was to determine how to 
evaluate this ‘‘what’’. Numerous methods were 
envisaged. 


Silent films, in colour, of carefully selected 
patients were used instead of actual patients, 
the examiner being replaced by a series of 
multiple choice questions concerning the 
patient presented. This method proved satis- 
factory and it is now in routine use by examin- 
ing bodies. 


Finally another method was found (pro- 
grammed testing) for evaluating the abilities 
of the intern when placed in a clinical sit- 
uation as real as possible and called upon to 
face the unforeseeable problems presented 
by every patient. 


In everyday routine the intern may be re- 
quired, for example, to see a patient who has 
just been admitted to the medical department. 
He goes to the patient, gets information from 
him and makes a clinical examination. He must 
then take a certain number of decisions. He 
calls for certain laboratory tests whose results, 
combined with those of the clinical examin- 
ation, will lead him to reach a diagnosis and 
decide on a treatment. The patient’s con- 
dition may then improve, worsen or remain 
unchanged by the treatment. The situation 
changes, new problems appear and fresh 
decisions must then be taken in the light of 
these new data. 


I Also called ‘Patient Management Problems”’. 


See also Simulation in instruction and evaluation in medicine 
in WHO Public Health Paper No. 61, Geneva 1974. 
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The programmed testing recreates, as far as 
possible, the changing situation represented 
by every patient. Each patient ts described 
in accordance with a real case history. From 
four to six “clinical problems”’ are presented 
following the case study with the aim of 
simulating a situation changing in time. The 
patient can be followed up for several days, 
weeks or possibly months, just as in real 
life, until he is discharged either cured or 
with his condition improved or, if he dies, 
passes to the autopsy table. 


At each step in time the candidate is required 
to make decisions; he immediately learns the 
results thereof and, with this fresh information, 
goes on to the following “‘choice’’, always 
concerning the same patient. 


The “eraser’’ technique 


The methodology of this type of test, as with 
programmed teaching, requires that the in- 
formation given to the candidate is hidden 
from him until he has made a decision and 
thus becomes entitled to obtain additional 
information. 


We shall not deal at length with the different 
technical difficulties that had to be overcome 
before a satisfactory system was found. As 
things are at present the appropriate infor- 
mation is hidden by a completely opaque 
layer of ink which can be removed, however, 
with an ordinary pencil eraser, or revealed 
by a system comparable to invisible ink (for 
the formula see page 4.48). 


The method can be easily used for examining 
a large number of candidates simultaneously. 


Examples of case histories 


A clinical observation, situation or study is 
described to the candidate and he is then 
asked: 


1. To study the details carefully and then 
the list of possible decisions presented for 
each ‘‘choice” linked with the initial ob- 
servation. 


2. To choose from the list only the num- 
bered items which seem important and 
appropriate. 
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3. To erase the corresponding opaque 
rectangle on the answer sheet, or to dis- 
cover the ‘“‘consequences’”’ by applying a 
special product. 


The candidate is reminded: 


1.20 Thateeextept in: rare’ instances, i as 
not suggested how many of the proposed 
decisions he should choose. 


2. That information (or “‘consequences’’) 
will appear in the space erased for both 
correct and incorrect choices. 


3. That since the information gradually 
revealed may orientate his subsequent de- 
cisions, he should consider them one after 
another in the order indicated. 


4n0 ButUthat owithin@eachy “choice: , “ithe 
order of the numbered decisions is pro- 
posed at random although it is advisable for 
the candidate to re-establish a logical order in 
his choice. 


Scoring 


The usual manual machine or computer 
method of scoring is employed, each space 
erased corresponding to one answer so that 
the candidate is unable to cancel a mistake 
once his choice has been made (the same 
applies in the case of a real patient). 


The candidate is penalised whenever he 
makes an incorrect choice and whenever he 
fails to make a choice which was appropriate. 
The scoring is thus negative, taking into 
account sins of both omission and commission. 


The choices proposed to candidates can be 
divided into three groups: 


(a) Appropriate; should be made with the 
aim of improving the patient’s condition 
(this is indicated by the mark +1); 


(b) Not indicated; should not be done and, 
if it is done, may be dangerous for the 
patient (mark -1); 


(c) Neutral, of debatable importance; may 
or may not be done according to local 


conditions, teaching, customs, etc. (mark 
0). 


The candidate who does not make a choice 
regarded as suitable by the examiner or who 
makes a choice regarded as not indicated or 
dangerous is penalised. 


Choice (c) has no effect on the scoring. 


Consequently, this is a scoring system com- 
pletely different from that for multiple choice 
questions where the candidate must select the 
best (and only) answer from several suggested. 


In programmed testing he must decide to 
select all those choices he regards as appro- 
priate for the treatment of the patient. He is 
not told how many choices he must make. 
The same applies in medical practice, where 
the physician makes a choice between what 
should be done and what should not be done. 
if he is proceeding on the right lines, he makes 
a certain number of decisions out of all those 
which could be made. 


Experience (immediate feedback after ‘‘era- 
sing”) gives him fresh data which will guide 
him towards new decisions. 


If he is on the wrong track, experience (‘“‘era- 
sing”) will show him his errors as they arise 
and give him a chance of changing his action 
although he will not be able to cancel out his 
mistakes. 


Improvement of case histories 


By means of test and measurement correlation 
studies the quality of questionnaires can be 
improved. The teachers who have drawn up 
the questions learn from the statistical study, 
question by question, how they can better 
test discriminatory qualities of judgment 
enabling a choice to be defined as ‘‘appro- 


2 6 ¢ 


priate’’, ‘‘non-indicated’’, or ‘“‘neutral’’. 


The task is different and considerably more 
arduous than that involved in drawing up the 
usual multiple choice questions. 


On the other hand, the examiners find them- 
selves on more familiar ground and feel that 
they are dealing with practical clinical sit- 
uations in a much more realistic way than 
when they had to decide on a single best 
choice. 


The method is far from perfect and calls for 
constant improvement, but gives new hopes 
for the evaluation of the clinical competence 
of physicians. It makes possible the evaluation 
of certain qualities which were not evaluated 
in the past, qualities considered essential for 
preparing the physician to assume independent 
responsibility in the practice of his profession. 
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example of a programmed test 


Specific objective: To deal in order of priority with several patients who come for treatment 


at the same time. 


Level required: To master the objectives 1, 5.1, 5.2 and 5.3 (see pp. 1.29 and 1.30). 


Description of the situation: 8. A two-month-old infant who has suffered 
Coming into the waiting room of achildren’s | from diarrhoea for a week and is obviously 
outpatient clinic, you find 15 children accom- dehydrated 

panied by their mothers, as follows: 9. A six-month-old infant, cyanotic, fever- 
1. A three-year-old with a scalp affection ish, and showing signs of dyspnoea. 

2A six-month-old infant suffering from 10. A five-year-old with expiratory brady- 
diarrhoea without outward signs of dehy- — penia and wheezing. 

saggee 11. A boy, 14 years old, with a phlegmon 
3. A newborn infant, 10 days old, with on his hand 

jaundice 12. A six-year-old girl who has suffered from 
4. A boy, eight years old, feverish abdominal pains for the last two weeks 

5. Agirl, three years old, withhyperthermia 13. An infant of seven months coming for a 
and dysphagia routine check up 

6. A mother carrying her newborn infant 14. An eight-year-old, pale but without signs 
under her veil of dyspnoea 

7. A 15-month-old girl with a cough and 15. A boy, seven years old, with arthritis of 
fever the right knee 

Section A 


You now decide to: (you are entitled to only one choice) 


Decisions to consider Consequences Mark 


1. Begin by examining the patients in the 
order of their arrival 


Five minutes after beginning your 
examinations, the nurse calls you into 
the waiting room. The condition of 
one of the children is critical. 


Select another decision. 


2. Have measurements taken of the 
temperature, weight and height of all 
the children. 


Meanwhile, one of the children suffers 
a respiratory arrest. Select another 
decision. 


3. Examine some of the children in 


Select the three children that you 
priority 


should examine first in your consulting 
room. Go on to Section B and follow 
up these three children among the 15 
on the list. 


4. Send children 7, 9 and 10 for X-ray 


While on their way, one of them faints. 


Select another decision. 
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Section B. Instructions: Select the children who need immediate attention from those 
numbered 1-15 described on page 4.44. 


Consequences Mark 


1. While you are examining this child, a child dies in the waiting room. 


2. You are called into the waiting room where a child is in convulsions. 


3. The nurse summons you urgently. 


| | 
— —_ 


4. In the waiting room, a child suffers a respiratory arrest. 


5. You are urgently called to the waiting room. 


6. Under the mother’s veil, you discover a newborn child 10 days old, 
cyanotic and congested. T: 95°F (35°C). Conjunctiva yellowish. Go on 
to Section C. 


7. You are summoned to the waiting room. 


\ | 
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GoW. 4.1 ke: 56.cm..CC:. 39cm, Ti: 96.4" 8 (35-870). 
Persistent abdominal skinfold; eyeballs sunken; cold hands and feet. 
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Go on to Section D. 


9. W: 7.6kg, H: 64cm, CC: 44cm, pulse 180/min, RF: 90/min. 
foci of crepitant sounds in both lungs. 


Go on to Section E. 


10. While making your auscultation you are summoned urgently. 


11. During your examination, the mother of another child bursts into your 
consulting room with her child who is in convulsions. 


— 


12. You are urgently called into the waiting room. 


13. You are urgently called into the waiting room. 
14. A child is in convulsions in the waiting room. 
15. Acchild has a respiratory arrest in the waiting room. 


W: Weight; H: Height; CC: Cranial circumference; T: Temperature; 


RF: Respiratory frequency. 
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Section C 
For this child you now decide to: 


Decisions to consider Consequences 


While you are doing this, the 
dyspnoea becomes more severe and 
the child becomes more congested. 


1. Interrogate the mother. 


During the examination, the child 


2. Make a complete examination of 
becomes cyanotic. 


the child. 


The moment the needle is inserted 
into the vein, the child has a 
respiratory arrest. 


3. Request biological tests. 


4. Immediately treat the symptoms. 


Section D 
For this child, you now decide to: 


Decisions to consider Consequences 


2. Make a complete examination. During the examination, the child 
becomes cyanotic. Pulse 180 


3. Request a blood count, sedimenta- While you are inserting the needle, the 
tion rate, urea and glycaemia child has a respiratory arrest. 


4. Immediately give emergency treat- 
ment. 


Section E 


For this child, you now decide to: 


Decisions to consider Consequences 


1. Interrogate the mother. While you are doing so, the child 
goes into convulsions. 


2. Make a complete examination. The child’s temperature rises to 
| 106.7°F (41.5°C) 


3. Perform a lumbar puncture. While you are doing so, the 
cyanosis suddenly increases. 


4. Immediately give emergency Go on to Section H 
treatment, 


4.46 


Treatment Section G 


ater [osetia 


1. Place the child near 


a heat source, 


4. Give oxygen 


5. Give a perfusion 


6. Give an enema of 
200 cc of water at 
14°C 


8. Cephalothin 


9. Ampicillin 


11. Ampicillin- 
gentamycin 


12. Penicillin 


13. Chloram- 
phenicol 


14. Hydrocortisone 
hemisuccinate 


pee ae 


Temperature rises to 
97.7 F (36.5) 


When done, the 


obstruction becomes 
less. 


When done, the 
cyanosis disappears 


Note under Section I 
the quantity of 
serum, the serum 
composition and 
rate of flow 


To bring down the 
temperature??? 
Cancelled by the 
officer on duty 


Prescription can- 
celled by the officer 
on duty 


On what basis? 
Prescription cancelled 


There is no valid 
reason to give this 
child antibiotics 


You have no valid 
basis at this stage. 
Prescription cancelled 
for the time being. 


for giving antibiotics. 
Prescription cancelled. 


This antibiotic is not 
for the newborn. 


Prescription cancelled. 


Not indicated. 


Prescription cancelled. 


You have no valid basis} -1 


+] 


+] 


resident physician 


When done: Temp: 
97.1°F (36.2°C) 


Child not congested 


Note under Section J 

the quantity of liquid, 
the nature of the per- 
fusion and the rate 


Treatment unsuitable, 
cancelled by the 


Prescription cancelled 
by the resident 
physician 


Not indicated. 


Prescription cancelled. 


Not indicated. 
Prescription cancelled. 


Not indicated. 
Prescription cancelled 


Prescription cancelled. 


Prescription cancelled 


Note the dose under 
Section J. 


To warm it up? 
It already has 
106.7°F (41.5°C) 
Treatment cancelled 


When done, the 
cyanosis disappears 


Note under Section 


K the quantity of 
liquid, the nature of 
the perfusion and 
the rate of flow 


Note under Section 
K the quantity and 
the nature of the 
liquid injected 


Note under Section 
K the dose and the 
route of adminis- 
tration 


Not indicated. 
Prescription cancelled. 


Not indicated. 
Prescription cancelled. 


Not indicated. 
Prescription cancelled. 


Note under Section 
K the dose and the 
route of adminis- 
tration. 


No indication. 


Prescription cancelled. 


Not indicated. 
Prescription cancelled. 
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Section I 


Section J 


Section K 


Drawn up by Professor J. -P. Grangaud, Test and Measurement Workshop, Timimoun, Algeria, February 1977. 


Note: 


Special printing techniques require somewhat 
sophisticated and often patented apparatus. A 
simple technique developed under WHO spon- 
sorship has been published under the title: 
An tmuisible ink process for use as an educa- 
tional tool. in Information: occasional publi- 
cation No. 3,1981,B.L.A.T. Centre for Health 
and Medical Education, Tavistock Square, 
London, WC1H 9JP. It is suitable for repro- 
duction by stencil or offset processes. The in- 
visible ink and the developer can be made 
from products easily obtainable on the market. 


*Rimoldi, 
1955. 


The simplest technique is still that described 
by Rimoldi* in 1955. The description of the 
clinical picture is typewritten on an ordinary 
sheet of paper; the ‘decisions to consider’ 
are also typewritten, but on cards of 
8 x 10 cm, and the “‘consequences”’ are on the 
back of the same cards. When the student has 
made his choice, he turns over the selected 
card and, if it is a certifying test, the teacher 
notes this. The P4 packs from H. Barrows, 
McMaster University, Canada, are a good 
example of the technique. 


H. J. A. A technique for the study of problem solving. Educational and psychological measurement, 15, 450 — 461, 
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Try to draw up a programmed test. Take as a basis a clinical observation or an epidemiological 
situation. Show the result to several colleagues and ask them for constructive criticism. 


List the advantages and disadvantages of this type of test. 
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stages of assessment 


tt prerequisite level testing 

a pre-testing 

oe interval testing 

=  ~=pre-final feedback comprehensive testing 
z» @ “subjective’’ impression 

a = final testing 

"a safety testing 


R follow-up testing 


different types of examinations 
during a course and their stages 


Prerequisite level test 


Before commencing a course it is necessary to 
ascertain whether the students have reached a 
certain level, namely the prerequisite level 
(refer back to page 1.44). A teacher must 
specify which knowledge he considers in- 
dispensable to ensure that the students 
assigned to him derive maximum benefit 
from the instruction he has planned for 
them. This test shows whether all the students 
are at this level or whether they are not, in 
which case coverage of this area must be 
ensured by modifying the proposed instruction 
to bring them up to this level. If this is not 
done the quality of instruction must suffer. 
Depending on the number of students who 
need bringing up to this level, the teacher 
must decide on the type of remedy — either 
reference to books or additional instruction — 
for the students concerned, possibly with the 
assistance of students who have reached the 
level and can be given the task of “‘instructor’’. 
As far as possible a “repeat’’ for all the 
students should be avoided, since this would 
amount to ignoring the diagnosis obtained by 
means of the prerequisite level test. 


Pre-testing 


When a given course commences it is advisable 
to make sure of the level of the students with 
respect to the course; on the one hand, this 
measurement of the starting level will permit 
the assessment of the real gain at the end of a 
course, on the other — and this has been shown 
experimentally — it may be found that some 
students are already quite advanced as regards 
the objectives envisaged for the course and 
allowance should therefore be made for this. 


This is a formative test (see pagess2 J 2.16). 


Interval testing 


These tests must be set as the course proceeds 
to give the student the feedback he needs in 
order to know where he stands after a parti- 
cular period of instruction. The teacher must 
see that these tests are, as far as possible, of 


the same difficulty as the final examination. 
One way of doing this is to select at random 
at least three ‘‘packets’’ from a group of 
questions. These three equivalent packets 
(a1, a2, 23, see diagram on page 4.51) will 
be used not only for interval testing but 
also for pre-testing (formative), pre-final feed- 
back comprehensive testing (formative) and 
final testing (certifying). Thus, when the 
student reaches the final examination he will 
not be haunted by the idea of its difficulty ; 
he will have been brought up to the necessary 
level beforehand. 


Pre-final feedback comprehensive testing 


This is a test of the formative type set before 
the final examination (comprehensive) of a 
course or the year (pre-final). Its purpose is 
to inform the student about his level of 
competence (feedback) and it should not be 
limited to a single subject but should cover a 
group of subjects. This will be facilitated if 
the school follows an integrated curriculum. 
If it does not, the teacher must include 
questions from other fields directly relevant 
to understanding of the subject taught. 


Subjective impression 


Evaluation of this type is carried out on the 
basis of the teacher’s personal knowledge of 
the students after contact with them during 
the year; he seeks to divide the students into 
three categories: good, average and bad. It 
would perhaps be preferable to divide them 
into two groups only: satisfactory and un- 
satisfactory. This evaluation should be carried 
out before the final examinations at the end 
of the year. 


Final testing 


These are of different types: oral, practical, 
traditional written, short open answer, or 
multiple-choice questions according to the 
educational objectives to be measured. They 
are organised after the end of a course. 


safety testing 


interval testing 5 
pre-final feedback 
comprehensive testing 


subjective impression 


final testing 


follow up testing 
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Safety testing 


This should be carried out if there is an ab- 
normal difference between the “‘subjective 
impression” and the results of the final 
examinations. If a student who was considered 
satisfactory or good has a bad mark in the 
final examination, it is essential to re-evaluate 
the situation and not to give the final examin- 


ation the role of final and arbitrary sanction 
which it has so often had in the past. 


Follow-up testing 


This is a form of evaluation which is carried 
out sometime after completion of the course 
to determine the extent to which the student 
has retained the acquired level of competence. 
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When several measuring instruments give consistent 
results despite different weaknesses, the reliability of 


the evaluation is increased. 
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desirable qualities of rating scales 


= clarity 
= relevance 


= precision 


= variety 
# objectivity 


= uniqueness 


factors influencing rating’ 


An evaluation made by a human observer is more or less objective and subject to error. The 
following are the factors influencing rating. The list is not exhaustive! 


Errors due to leniency 


Leniency is a well known factor. One means 
of counterbalancing this tendency is to use a 
scale containing only one ‘‘unfavourable”’ 
appraisal in five, for instance: 


Sa ee eee Lee See eee 
Poor Average Good 


Very good —_— Excellent 


In this case the appraisals will probably be 
distributed symmetrically around ‘‘good”’. 


Central tendency 


Examiners have a tendency not to give 
extreme appraisals and hence to group all 
candidates around the mean. This central 
tendency may be reduced by using a scale 
that is wider at the centre than at the ends, 
for example: | 
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The halo effect 


One particular feature of a candidate some- 
times seems so important to the examiner 
that it influences the overall evaluation. 
Thorndike called this the halo effect. However, 
this effect is reduced as the number of separate 
aspects of the problem dealt with by the eval- 
uation is increased. 


The logical error 


The logical error is similar to the halo effect 
and occurs when the examiner supposes that 
there is a relationship between two variables 
to be evaluated and that “if the first variable 
is of a particular order, the second will be 
similar’. This error may be reduced if the 
evaluation relates to an observable element 
rather than to an abstraction which could 
lead to semantic confusion. 


The contrast error 


An observer who is very orderly will tend to 
consider, by contrast, that other people are 
less orderly than he is, and vice versa. On the 
other hand, people frequently believe that 
“others are like me’ and are very surprised 
to see that this is not so. 


The proximity error 


If an observer evaluates two different factors, 
the evaluation of one factor tends to influence 
that of the other, and the shorter the interval 
between the two, the more pronounced the 
tendency (proximity error) will be. 


*Guilford, Psy chometric methods, pp. 278—280. 


test construction specification table 


Content Number of test items in relation to 
areas 
type of competence measured by test 
Recall of | Interpretation | Problem 
facts of data Solving 
oo 1 i 


= oe 


Complete this specification table, making a qualitative analysis of the exercises proposed in this 
guide: for each exercise, decide which is the level tested — Level 1: Recall of facts; Level 2: 
Interpretation of data: Level 3: Problem solving; (reread page 1.38). Then calculate ther per- 
centage share of each level in the total. Check your results on the next page. 


Competence measured Number of 
by the test tests 
Finca ti rip Percent- 
Objective escrip ion Oo t € exercise Recall Interpretation Problem per per age 


(for details, see references page 15) of Facts} of Data Solving exercise Ichapter 


Identifying professional activities 
Listing the main functions of a 
category of health personnel 


3 
3 


20/26 | Analysis of the relevance of a 
programme 
Identifying the components of a 
professional task 


Selecting active verbs corresponding 
to a task 
Identifying the elements of a task 


Identifying the elements of an 
educational objective 


Chapter 1 


Chapter 2 


Drawing up specific educational 
objectives 


Drawing up contributive educational 
objectives 


Critical analysis of an educational 
objective 


1to9 | Evaluation of intellectual skills with 
regard to educational objectives 


17 Specification of educational 
decisions 


Distinguishing between formative 
and certifying evaluation 


oma 
NO 


13,14,16 | Choice of evaluation method 
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Objective 


10 to 18 


Description of the exercise 


Comparing different methods of 
evaluation 


Diagram showing an evaluation 
system 


Description of obstacles and tactics 
in the implementation of an 
evaluation system 


Evaluation of intellectual skills 
with regard to evaluation planning 


Description of the teacher's 
functions 


Comparison between several teaching 


methods 


Construction of an organisational 
chart for programme implemen- 
tation 


Description of obstacles to and 
tactics for setting up a new 
programme 


Evaluation of programme 
construction skills 


Listing advantages and limitations 
of evaluation by students 


Construction of a practical test or a 


project 


Constructing a rating scale for 
attitudes 


Preparation of a written question 


Preparation of short, open answer 
questions 


Composition of multiple choice 
questions (MCQ) 


Competence measured 
by the test 
Recall |Interpretation|Problem| per per 
of Facts} of Data | Solving jexercise| chapter 


Number of 
tests 


faa Chapter 2 


Chapter 3 


Chapter 4 


Percent- 
age 


Competence measured Number of 
by the test tests 


Recall |Interpretation|Problem| per 
of Facts} of Data | Solving jexercise 


Percent- 
per age 


chapter 


Description of the exercise 


Objective 


36 ~—-|_: Construction of a programmed 
examination 
36 Completion of a specification table 
for an examination 
— 39 Calculation of the minimum pass 
level for MCQ tests 
40 Calculation of the difficulty and 


discrimination indexes of a 
question 


Number of 
questions 


es 
a 


Check your answers to the exercise on the preceding page 


Your percentages should be about equal (within 10%) to those given below. Broadly, you should 
have found about 20% of the questions/tests at level 1 (recall of facts) and 80% at levels 2 or 3 
(above level 1). At least that is what the author of this Handbook believes. 


Exercises 
For the exercises marked with an asterisk, you will find answer checklists in the Handbook. 
e 15 for the page numbers. 
meio ‘ Competence measured Number of 
by the test tests 


Recall || ion|Probl pple 

= aie . terpretation|/Problem| per per age 

ne f the exercise dec 8 pi i ; 

Objective ee of Facts} of Data | Solving |exercise| chapter 
Identifying professional activities ee oe 
Listing the main functions of a 
category of health personnel 


20 to 26 | Analysis of the relevance of a 
programme 


Identifying the components of a 
professional task* 


Selecting active verbs corresponding 
to a task 


Identifying the elements of a task * 


Identifying the elements of an 
educational objective* 


Drawing up specific educational 
objectives 


Drawing up contributive educational 
objectives 


and certifying evaluation* 


13,14,16 | Choice of evaluation method 


See 


Critical analysis of an educational 
objective 
1 to 9 Evaluation of intellectual skills with 7 
regard to educational objectives* 
17 Specification of educational 
decisions & 
. . . . _ 
12 Distinguishing between formative - Si a 
co 
A ony 
nae : 
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Exercises 


For the exercises marked with an asterisk, you will find answer checklists in the handbook. 
See page 15 for the page numbers. 


10 to 18 | Evaluation of intellectual skills 
with regard to evaluation planning* 


Competence measured Number of 
by the test tests 


per 
chapter 


Recall |Interpretation|Problem 
of Facts} of Data {Solving 


per 
exercise 


Description of the exercise 


Comparing different methods of 
evaluation 


Diagram showing an evaluation 
system* 


Description of obstacles and tactics 
in the implementation of an 
evaluation system 


Chapter 2 


Description of learning situations 


Description of the teacher’s 
functions 


Selection of teaching methods 


Construction of an organisational 
chart for programme implemen- 
tation* 


Description of obstacles to and 
tactics for setting up a new 
programme 


tie 


19 to 29 


34 


Evaluation of programme 
construction skills* 


Listing advantages and limitations 
of evaluation by students 


Chapter 4 


Construction of a practical test or a 
project 


Constructing a rating scale for 
attitudes 


Preparation of a written question 


Preparation of short, open answer 
questions 


Composition of multiple choice 
questions (MCQ) 
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Exercises 
For the exercises marked with an asterisk, you will find answer checklists in the handbook. 


Competence measured Number of 
by the test tests 
Recall |Interpretation|Probl Bred 
ne Description of the exercise ecall jInterpretation|Problem) per | per BO - a: 
36 Construction of a programmed 
examination 

Completion of a specification table 

for an examination* 

Calculation of the minimum pass 

level for MCQ tests 

Calculation of the difficulty and | 

discrimination indexes of a 

question* 
30 to 40 | Evaluation of test and measurement 

Boe ee aed 


ee ee 
—————— eee 


Number of 
questions 73 


pee 
Percentage Deiies tiem racy 


See page 15 for the page numbers. 


Chapter 4 
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relative and absolute criteria tests 


These two expressions are also referred to in 
the literature as norm-referenced and criterion- 
referenced tests. 


It is very important to distinguish between 
tests based on reference to the ‘“‘norm’”’ (i.e. 
in accordance with the curve for the results 
of all the students who have taken the same 
test, and that is why this criterion is termed 
relative), and tests based on reference to a 
“criterion”’ (i.e., in relation to the description 
of an acceptable performance, that is to say, 
the specific educational objective fixed in 
advance). 


An absolute criteria test is one deliberately 
designed to give results that can be directly 
interpreted in terms of the acceptable level 
of performance of the person tested. It 
enables a person’s performance to be evaluated 
in relation to a previously specified level of 
performance. The aim, therefore, is to 
determine whether a person has or has not 
mastered a particular task, and not to compare 
one person’s performance with that of another 
or of a group of persons. A relative criteria 
test, on the other hand, aims at enabling a 
valid discrimination to be made between 
persons on the basis of different types of 
performances; it is thus a competitive test. 


Relative criteria tests are the ones most fre- 
quently employed for examination purposes. 
Unfortunately their disadvantages greatly 
outnumber their advantages, for if a group of 
students is particularly brilliant the utilisation 
of relative criteria tests will lead to some of 
them being failed although their level of 
performance may be satisfactory from the 
absolute viewpoint. On the other hand, if a 


given group of students has on the whole a 
low performance level (because the appro- 
priate instruction has not been given, or has 
been poorly given, or for some other reason) 
the relative criteria system may allow “poor” 
students to pass if they are above the average 
of their group. The consequences may be ex- 
tremely harmful for the health of the pop- 
ulation. 


If, on the other hand, a situation arises in which 
a certain number of persons have to be selected 
for admission to a given course of study, it 
becomes necessary to compare their perfor- 
mances. In that case a relative criteria test is 
appropriate. 


If it is felt unanimously by an evaluation 
committee, for example, that all the students 
should be able to master an emergency pro- 
cedure, then this can be ascertained only by 
an absolute criteria test. These tests are, 
indeed, the only ones that justify the certifying 
of any health worker as having demonstrated 
an acceptable level of performance. 


It is thus theoretically possible, and even 
desirable, that all the students taking an 
absolute criteria test should ‘‘pass’. That 
would demonstrate the high degree of effec- 
tiveness of the training programme. It would 
also, of course, be theoretically possible for 
all the students to fail. 


On the other hand, a relative criteria test is, 
by its nature, one which will always divide 
the students taking it into at least two cate- 
gories, those who succeed and those who do 
not, without any guarantee that the former 
are also competent. 


pa me od a a ew a a aw Ea J 
It basically comes down to a choice between a measure- 
ment strategy which compares people versus one that 
lets us know what it is that people can or cannot do. 


(J. Popham) 
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A measuring technique adapted to absolute criteria 
tests is suggested on pages 4.62 — 4.63, namely cal- 
culation of the acceptable level of performance (or 
minimum pass level). You will find on pages 4.65 — 
4.70 the measuring techniques suitable for relative 


criteria tests (difficulty and discrimination indexes). 
wees oo sod oo eon oho poo 


4.61 


calculation of the acceptable level of 
performance (ALP) for a MCQ test 


1. Definition 


The acceptable level of performance is a thres- 
hold value making it possible to decide 
(according to absolute criteria) whether a 
student ‘“‘who knows barely enough’’ should 
be passed or failed. 


Calculation of the ALP for a test is not valid 
unless the number of MCQ is more than 30. 


Use of the ALP involves an advance judgment 
(before the test) on the relative difficulty of 
each question and enables a judgment based 
on the test as a whole to be made. 


Calculation of the ALP depends on the 
collective decision of several teachers each of 
whom has first made an independent judg- 
ment. 


2. Procedure* 


To calculate the acceptable level of perfor- 
mance (ALP) of a student for a MCQ test: 


2.1 the evaluation board decides what 1s 
the correct answer to each MCQ; 


if all the choices are equivalent, the index 


if one choice must be eliminated, the index 
if two choices must be eliminated, the index 
if three choices must be eliminated, the index 


if four choices must be eliminated, the index 


2.2 the board decides which answer or 
answers must definitely be eliminated 
by the student, other than by chance; 


2.3 the board calculates the acceptability 
index for each MCQ; 


24 the ALP for the test as a whole is the 
sum of the acceptability indexes for 
each MCQ. 


The acceptability index for a MCQ is cal- 
culated as follows: 


Carefully study all the choices offered and 
decide which the student “who knows just 
enough to pass” should be able to reject. For 
example, if a question offers five choices (only 
one of which is the correct answer) and it is 
deemed that the student “who knows just 
enough to pass” should be able to reject one of 
these choices straightaway, it follows that the 
marginal student could obtain the correct 
answer by mere chance approximately one 
time out of four. In this case the acceptability 
index of the question 1s 0.25. 


with five with four 
choices choices 


Let us take two MCQ which are worded identi- 
cally but where the choice of answers is 
different. 


Question: 


Which of the following values corresponds to 
the number of red cells per mm? of blood in 
a healthy adult? 


*This procedure was developed from an article by L. Nedelsky 
(Absolute grading standards for objective tests. Educ. 
Psycholog. Meas., 14, 3-19, 1954). 
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1 2 
A 500 000 A 4000 000 
B_ 1.000 000 B 4500 000 
C 2000 000 C 4750 000 
D 3000 000 D 5 000 000 
E 5 000 000 E 5 250 000 


In case 1, an acceptability index of 1.00 
could be considered while in case 2 it could 
be 0.25. 


3. Comments 


The ALP has little value if it is not based on a 
detailed analysis of each of the questions in a 
test, including consideration of incorrect 
choices just as much as of correct answers. 


The validity of the estimate of the ALP also 
depends on obtaining independent judgments 
from several teachers who have paid attention 
to the educational objectives and the level for 
which the examination is intended. The qual- 
ity of the estimate will be the greater the larger 
the number of teachers involved. 


When the differences between the judgements 
obtained are relatively small, the extremes 
can serve to define a “‘grey zone’’ below which 
the results will be regarded as distinctly ina- 
dequate (failure) and above which the results 
will clearly indicate a success. For example, 


if the mean of the estimates of one teacher 
for the ALP of a test is 43% whereas two other 
teachers obtain figures of 45% and 47%, 
respectively, then it could be recommended 
that any score below 43% should be regarded 
as a failure, that any score above 47% be 
regarded as a success, while a score between 
43% and 47% should be regarded as being in 
a grey zone. It would remain to be defined 
what should be done in the latter case. 


If the differences between the judgments 
obtained by several teachers are large then 
the criteria of the educational objectives 
should be revised. 
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New ..... calculate the acceptable pass level for all the MCQ you drew up on page 4.39, 
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significance and form of communication 
of the results of a test as a whole 


Relative Absolute 
Criteria Tests Criteria Tests 
Base distribution of educational objectives 
| scores 
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Pass Score a posteriori a priori | 


— alphabetical or 


alphabetical ; 
numerical scoring 


or numerical 
scoring — descriptive profile 
of student 


Form of 
Results 


Meee oe OO Ole Oo eee OO 0 Ghee 


If we have objectives that are important to achieve, that i 
is, if there is a genuine need for them to be achieved 
because a meaningful consequence would result in 
achievement or non-achievement, then it is also import- 

ant that we find out if that achievement was accom- 
plished. We cannot do so by comparing one student 

with another, we can only do so by comparing the 
performance of each student with the performance 

called for by the objective. 


Mager 
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steps in item analysis 
(relative criteria tests) 


1. award of a score to each student 

2. ranking in order of merit 

3. identification of groups: high and low 

4. calculation of the difficulty index of a question 


5. calculation of the discrimination index of 
a question 


6. critical evaluation of each question enabling a 
given question to be retained, revised or rejected 


4.65 


Li 


A practical, simple and rapid method is to 
perforate on your answer sheet the boxes 
corresponding to the correct answer. By 


pom 
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Award of a score to each student 
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placing the perforated sheet on the student's 
answer sheet the raw score (number of correct 
answers) can be found almost automatically. 
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2. Ranking in order of merit 


Assuming that the scores of 21 students have 
been obtained (alphabetical list on the left), 
this step consists merely in ranking (listing) 
students in order of merit (in relation to 


A 


Albert 
Alfred 
Andrew 
Ann 


Brian 
Christine 
Elise 
Emily 
Felicity 
Frances 
Frank 
Fred 
Harriet 
lan 


John 


Jennifer 


Margaret 
Michael 
Patrick 
Peter 


Philip 


the score) proceeding from the highest to the 
lowest score. Let us assume the list as under A 
and then rank the students to obtain distri- 
bution B, ranging from 4 to 27. 


B 

1 Patrick 

2 Frank 

3 Ann 

4 Emily 

5 Jennifer 

6 Christine 

Vi Andrew 

8 Elise 

9 Jan 
10 Fred 
j Brian 
12 Felicity 
13 Margaret 
14 Philip 
15 Frances 
16 John 
L7 Alfred 
18 Harriet 
19 Michael 
20 Albert 
21 Peter 


3. Identification of high and low groups 


Ebel! suggests the formation of “high’”’ and 
“low” groups comprising only the first 27% 
(high group) and the last 27% (low group) of 
all the students ranked in order of merit. 


Why 27%? Because 27% gives the best com- 
promise between two desirable but contra- 
dictory aims: 


1. making both groups as large as possible; 


2. making the two groups as different as 
possible. 


Truman Kelley showed in 1939 that when 
-each group consists of 27% of the total it can 
be said with the highest degree of certainty 
that those in the high group are really superior 
(with respect to the quality measured by the 
test) to those in the low group. If a figure of 
10% were taken, the difference between the 


two means of the competence of the two 
groups would be greater but the groups would 
be much smaller and there would be less 
certainty regarding their mean level of per- 
formance. 


Similarly, if a figure of 50% was taken the 
two groups would be of maximum size but 
since the basis of our ranking is not absolutely 
accurate, certain students in the high group 
would really belong to the low group, and 
vice versa. 


While the choice of 27% is the best, it is, 
however, not really preferable to 25% or 
33%; and if it is preferred to work with} or + 
rather than with the somewhat odd figure of 
27% there is no great disadvantage in so doing. 


For the rest of our analysis we shall use 33%. 


4. Calculation of the difficulty index of a question 


Difficulty index 


Index for measuring the easiness or difficulty 
of a test question. It is the percentage (%) of 
students who have correctly answered a test 
question; it would be more logical to call it 
the easiness index. It can vary from 0 to 100%. 


Calculation 


The following formula is used: 


Difficulty index = —~+* x 100 
where H = number of correct answers in 
the high group 
L = number of correct answers in 
the low group 
N = total number of students in both 
groups 


(Do exercise on page 4.71). 


5. Calculation of the discrimination index of a question 
Discrimination index Calculation 
The following formula is used: 


An indicator showing how significantly 4 Discrimination index = 2 x (Hi - L) 
question discriminates between ‘‘high’’ and 2 


“low” students. It varies from -1 to +1. (Do exercise on page 4.71). 


1 ° 
Ebel, R. L. (1965) Measuring educational achievement, Prentice Hall, pp. 348 — 349. 
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6. Critical evaluation of a question 
This is based on the indexes obtained. 


Difficulty index: the higher this index the 
easier the question; it is thus an_ illogical 
term. It is sometimes called ‘‘easiness index’”’, 
but in the American literature it is always 
called ‘difficulty index’’. 


In principle, a question with a difficulty 
index lying between 30% and 70%* is accept- 
able (in that range, the discrimination index 
is more likely to be high). 


If for a test you use a group of questions with 
indexes in the range 30% — 70%, then the 
mean index will be around 50%. It has been 
shown that a test with a difficulty index in 
the range of 50% — 60% is very likely to be 
reliable as regards its internal consistency or 
homogeneity. 


Discrimination index: the higher the index 
the more a question will distinguish (for a 
given group of students) between “high” 
and “‘low’’ students. When a test is composed 
of questions with high discrimination indexes, 
it ensures a ranking that clearly discriminates 


between the students according to their level 
of performance, i.e., it gives no advantage to 
the low group over the high group. In other 
words, it helps you to find out who are the 
best students. 


It is most useful in preparing your question 
bank. Using the index**, you can judge 
questions as follows: 


0.35 and over 


Excellent question 


0.25 to 0.34 Good question 

0.15 to 0.24 Marginal question — 
revise 

under 0.15 Poor question — most 


likely discard 


*Some authors give values between 35% and 85%. 


** Remember that the index has an indicative rather than an 
absolute value. 
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uses of indices 
aim: review of questions 


» Difficulty Index 
diff. ind. 


recommended 
(in order to obtain a 
good discrimination index) 


P Discrimination Index 


dis. ind. = 


Discard Revise Excellent 


(most likely) 


Given a group of 21 students (see page 4.67). Using 33% of them to constitute a high group of 
7 and a low group of 7 (33% of 21), the following table shows the answers given by those two 
groups (high and low) to 10 multiple choice questions (numbered from 1 to 10 in the first 
column). The correct answer for each of those ten questions is given correspondingly in the 
second column. In the 14 consecutive columns are shown the answers given by each student 


to each question. 
H+L DIF. IND. 


Ranking in order of merit 


Patrick 
Frank 
Jennifer 
Christine 
Andrew 


Ann 
Emily 


Question | Correct 
No. Answer 


Bak FRE Eo eee 
COCCCCAC 


High group 


CEEECBA 
CB-CDBA 


ee Be PBR BR BBB BBBEEE. 7 h4 1 ult. 1.3 

2 CO VeMO CC Gr TC Cerr cer eats ip 

3 A BABA BBAIBBABAA ALA 4) o7 let 
4 ECE EGE CRP Oe DAE tich eh caer os 

5 Boo et BiBsC Ee BGC B EE EE 9,0 bogs 

6 Dp |DDDCDDDIEDEEDEE 

7 phd Geshe eM malin een AR eat 

8 c |BBBCBCBIBBBBBCC 

g E | 

10 c 


Now try to: 
1. Calculate H - L for questions 6 to 10. 


2. Calculate the difficulty index and the discrimination index for questions 1 to 5. 


Check your results on the next page. 
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Check your answers 


| Ranking in order of merit 


Correct 
Answers 


Question 
No. 


Patrick 
Frank 
Emily 
Jennifer 
Christine 
Andrew 


Ann 


BBBBEEE 
Cccccce 
BBABAAA 
ECCAE-C 
He Opa ed we hd 
E-D767E-Dp ESE 
CCCCCCC 
BBBBBCC 
CEEECBA 
CB-CDBA 


BBBBBBB 
Cccccccec 
BABABBA 
BsbC ESC; Bebeu 
BCBBCEC 
DDDCDODD 
cccccce 
BBBCBCB 
EXEVESE*E SESE 
CCCCCAC 


om 
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Conditions for the application of this procedure for item analyses, in particular: 


1. it applies to re/ative criteria tests (the procedure leads to a choice of questions that 
tend to maximise variance and ensure discriminatory ranking), 


2. itis applicable only to questions scored dichotomously (1:0), 


3. it should not be applied if the total number of students is very small (a minimum of 
20 students could be proposed as a “pragmatic” criterion). 


Question analysis card These cards as a whole will constitute the 


To facilitate the construction of a question Oe 
bank it is advisable to enter the statistical The front and the back of the card of this 
results for each question on a separate card. type could be as follows. 


Front 


= 
Course 
of test 


. 
hil 
o 
Cin Medcid | mca. = ‘ 
bree 
+ 


MCQ 


aoe 

2nd year MCQ a | Bee 0.61 

| oe 
Speciality 
Board 


Back 


Endocrine System Nature of question: MCQ 


Objective tested Ability to explain the physiological functioning of the thyroid gland 
eee mere | Intellectual skills — Level 1 (recall of facts) 


Question Which of the following produces an increased secretion of thyroid hormone 
in a normal subject? 


Answers 


Administration of TSH 


Administration of thiocyanate 


A 

B 

G: Administration of propylthiouracil 
D Administration of thyroxine 
E 


Some other treatment 


Sternberg, Chapter 2, page 112 — prepared by Mr. X in February 1972 


4.73 


Personal Notes 
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Check your results on page 4.82. 


Question 1 


The administration of a test before the beginning of a learning period (formative pre-testing) has 
the following advantages except one: 


A. To modify educational objectives of that period. 

B. To provide ways for less well prepared students to catch up. 
C. To modify the required pass level (mark). 

D. To provide a base from which to measure real progress. 

E. To exclude weak students from the learning period. 
Question 2 


All the following stages, except one, are recommended for scoring tests of the ‘‘essay”’ type: 


A. Write the elements of the answer for each of the questions asked. 

B. Correct the answers question by question rather than student by student. 
C. Determine the pass score on the basis of a sample of answers. 

D. Correct the answers while preserving the anonymity of the students. 

= Identify three levels only: honour, pass, fail. 

Question 3 


The content validity of a written test is usua//y obtained by means of: 


A. 


Me Se Se 


Collective and carefu! review of the questions. 
Pearson’s correlation coefficient. 

Factor analysis. 

An “inter-rater” reliability coefficient. 


A mean discrimination index. 
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Questions 4 to 6 


A test with 50 questions is administered to a group of 45 students. There is a choice of five 
answers to every question. Only one of these choices is the correct answer. One point per 


correct answer is allocated in calculating the total score. 


Question 4 


Assuming that none of the students have any knowledge of the test subject (i.e. they choose 
their answers by guessing), which of the following will be closest to the mean score of the 


group? 


A. 0 
BraeS 
Gav 
Des 15 
Eee 20 
Question 5 


On dividing this group of 45 students into 3 groups of 15 each, on the basis of the total score of 
each student, it is found that, for the first question, nine students out of 15 in the high group 
and three out of 15 in the low group have given the right answer. For this question the 
difficulty index is: 


A. 12% 
B. 27% 
C.. 30% 
D. 40% 
22-60% 
Question 6 


Under the same conditions, the discrimination index is: 


A. 0.12 
Bi 27 
CG. 0,30 
D. 0.40 
E. 0.60 
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Question 7 


On the basis of these indexes, which of the following decisions would you take concerning this 


question? 

A. It should be discarded from-the question bank. 

B. Itshould be referred to a drafting committee for revision. 
C. It should be retained in the bank as it is. 
D 


A decision other than A, B or C. 


Questions 8 and 9 - 


The following data concern a multiple choice question set to 300 students, the correct answer 


being D. 


Choice of answers 


Cee 


Question 8 

These data iow that: 

A. _ half the students answered the question correctly; 
B. all the distractors were of good quality; 

C. the question was of high validity; 
D 


the question was not very relevant. 


Question 9 

in view of these data, the examination board may decide: 

A. that this question should be reviewed since it is insufficiently discriminatory ; 
B. that this question should be discarded from the question bank; 

C. that this question is of low validity; 
D 


none of the above. 
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Question 10 
What could generally be expected on doubling the length of a test whose mean discrimination 
index is 0.52 (by adding questions more or less equivalent to the previous ones) ? 


A certain increase in the reliability and the validity of the test. 
Only a certain increase in the reliability of the test. 
Only a certain increase in the validity of the test. 


A certain decrease in the reliability and validity of the test. 


moo ® 


No effect on either the reliability or the validity of the test. 


Questions 11 to 16 


Use the following key in answering this series of six matching type questions: 


traditional oral test 


B = written test of the essay type 
C = so-called written ‘‘objective’’ test (MCQ) 
D = standardised practical test, or written and oral simulation tests (programmed 


examination) 


Indicate the type of test most su/tab/e for evaluating each of the following performances: 


Question 11 


Recall of concepts. 


Question 12 


Ability to solve problems. 


Question 13 


Ability to communicate satisfactorily with the patient. 


Question 14 


Verbal expression. 


Question 15 


Skill in examining the patient. 


Question 16 


Ability to make a synthesis. 
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Question 17 


The system of “‘relative”’ criteria of competence implies the following consequences, 
except one. Which? 


A. Leads to an embarrassing disagreement among those responsible for applying the resultant 
decisions. 


B. Leads to the failure of certain students ina particularly competent group. 


Enables one group to become the arbiter of the standards according to which it is 
judged. 


D. Enables “low group” students, who are however superior to the mean of the whole group 
to which they belong, to pass. 


E. Creates an arbitrary fluctuation in the desirable level of competence at a given moment. 


Questions 18 and 19 


The author of the following multiple choice question was asked to establish its acceptability 
index. 


The diameter of a normal erythrocyte (according to Wintrobe) expressed 
in um (microns) is equal to: 


A. 45 | C575 Dx 835 
B. 65 E. 10.5 


He felt that a student who “‘knew just enough to pass’ should be able to reject right away 
choices A and E. 


Question 18 


Indicate which among the following values of the acceptability index corresponds to the 
author’s choice: 


Aor aut0 C:.-0.25 D:=-0:33 
62. 7-0,20 E. 0.50 
Question 19 


lf the item C was not included, what then would be the acceptability index? 
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Question 20 
According to the theories about absolute or relative criteria tests, all the following statements 
are correct except one. Indicate which is false: 


A. The calculation of the discrimination index provides a statistical datum applicable to 
absolute criteria tests. 


B. The calculation of the acceptable level of performance (ALP) of a test is applicable to 
criterion-referenced tests. 


C. The acceptable level of performance (ALP) of a test is equal to the sum of the acceptability 
indexes of each question. 


D. The value of the difficulty index influences the value of the discrimination index. 


Bs eb GON erg ah Kee Of g Py By me EF WG fy Png Pa a Se fo ge A 


Performance assessments designed to measure compe- 
tence for a job or task are inescapably imperfect because 
of measurement errors, and because task components 
can never represent the total job. 


Se WM gi PEA Bf fF cy PY ow BY a GS a A 


Se 1 QA ee oe Fc as Ct KP ae fae a J Fg) es 3 eM 


If you find this Handbook too complicated you may 
wish to refer to another publication of the World 
Health Organization: 7eaching for better learning: A 
guide for teachers of primary health care staff by 
F. R. Abbat, WHO, Geneva, 1980. 
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please 


Aas if you know how to define specific 
objectives 


Rees if your colleagues turn green with 
envy on reading your criteria 


Retake if you are able to choose the most 
suitable teaching technique 


aoa 8 if you can put it into practice 
ee gee. if you are a leader 

Ba ece, if your students admire you 
Be ee 8 if your examinations are valid 
eee if your scores are objective 
do not be influenced by all this. . ! 


... and over-estimate the importance of 
your own subject. 


do not forget relevance .. ! 
the relationship between your teaching 


and the institutional objectives derived 
from community health needs. 


Answers suggested for the exercise on pages 4.75 — 4.80. 


Suggested If you did not find the correct answer, consult 


ould answer the following pages again 


2.15 and 2.16, 4.49 — 4.52 


oo 


4.28 
2.33 


use basic rules of arithmetic 


4.65 — 4.72 


oO OAaOn oO FP W N 


2.36 and 2.37 


2.22, 2.30 and 2.31, 4.22 — 4.40 


4.61 and 4.62 


4.62 and 4.63 


E 
C 
A 
C 
D 
D 
C 
A 
D 
A 
C 
D 
D 
A 
D 
B 
A 
D 
= 
A 


4.61 — 4.63 
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workshop 


how to organise an educational 


workshop 


The purpose of this chapter is to facilitate 
the task of anyone who wishes to prepare 
and run an educational workshop. 


It contains the basic documents (or refers 
to the documents) required to organize a 
very short workshop (lasting 3 to 4 days), 
known as a mini-workshop. 


The chapter obviously makes no claim to 
qualify the reader to organize all possible 
types of educational workshop regardless of 
the educational philosophy involved. The 
type of workshop proposed is designed to 
use the WHO Educational Handbook as its 
source of theory. 


Nevertheless, most of the general principles 
involved are also valid for longer workshops. 


This chapter responds to a need often 


expressed by those attending workshops run 
by WHO: the wish to pass on knowledge of 
the systematic approach to educational 
problems to their colleagues by means of a 
short meeting. 


The method proposed itself uses the systematic 
approach. It also stresses educational principles 
such as the following: 


allowing the participant to prepare and 
select the objectives to be reached will 
increase his motivation; 


giving the participant an active role will 
make teaching more effective; 


providing the participant with regular 
opportunities to see the progress he is 
making will increase his learning speed 
and improve the quality of the know- 
ledge and skills he acquires. 


BGalaG) fesse be ER ete Eee ee Bee eel oe ae ees 
Attention! 


This chapter has been prepared for teachers who have 
attended at least one educational workshop and thus 
know how such meetings operate and are familiar with 


the Educational Handbook 


Fe gegetay af alapagale iets chepobateuiahelalntefenalsi= ps 


This chapter is designed for use in a wide 
range of local contexts. This means that any 
user should always remember to make 
allowance for the cultural, educational and 
occupational background of participants. 
Modifications may therefore be contemplated, 
worked out, put into effect and, most 


3.02 


important, evaluated. However, a user with no 
special training in educational science is not 
advised to introduce significant changes right 
away. It is better to use the proposed system 
as it stands and be prepared to amend it in the 
light of experience (particularly as a result 
of “feedback” from participants). 


what the term workshop implies 


A workshop is a meeting during which 
experienced people in responsible positions 
come together with experts and consultants 
to find solutions to problems that have 
cropped up in the course of their work and 
that they have had difficulty in dealing with 
on their own. Participants themselves select 
the objectives they wish to reach and help 
in choosing the problems for group work.* 


An essential feature of the workshop is 
complete active involvement by each 
participant: the whole point of attendance 
is to work and to learn from practical 
experience. 


One of the commonest methods used in 
workshops is group discussion of selected 
problems, the size of the group being small 
enough to encourage full participation by 
each member and large enough for each 
member to gain from the experience of the 
others. There is nothing magical about a small 
group but it does offer each member an 
opportunity to make his own contribution. 
It gives participants the chance to discuss and 
solve the problems of greatest interest to 
them. The fact that each member can find 
something in the experience of others that has 
a bearing on the questions of most interest 


to himself will make his work more 
meaningful. 


The workshop method makes _ everyone 
(organizers and participants) responsible for 
helping to find solutions to the problems 
selected. Participants may have to act as 
group leaders or rapporteurs. The organizers 
are generally there to be consulted by 
participants and to help them where necessary 
(not to give lectures or impose solutions). 


The workshop programme makes provision 
for plenary sessions, discussions in small 
groups and other activities but does not 
follow a strict hour-by-hour timetable. On 
the contrary, the programme adapts itself to 
the way the work is going. For example, a 
plenary session will be held whenever there 
is a need to pool the results of group dis- 
cussion, to clarify a point for all participants 
or to introduce some activity that requires 
unanimous approval. 


A mini-workshop is a short workshop (lasting 
three or four days in the case described in 
this document). 

*These problems and objectives are occasionally selected 
(in most cases by correspondence) before the workshop 


begins, to allow the participants time to prepare for it. 
However, this does not apply to a mini-workshop. 


aims of an educational mini-workshop 


The workshop aims at introducing participants 
to a systematic approach to educational 
problems. They must, so to speak, not only 
be made “hungry for more’”’ (i.e. anxious to 
carry on learning about the subject and 
increasing their skills) but also be given “‘food 
for the journey” (documentation containing 
enough references to enable them to make 
progress after the workshop is over). 


The workshop aims at stimulating a given 
proportion of participants to wish to reach 
at least the objectives set out in the 
Educational Handbook (see pp. 12 and 13) in 
the course of the ensuing year. 


It has been found from experience that some 
participants may go far beyond these aims 
and embark on activities such as: 


— defining X% of the specific objectives for 
the subject they teach; 


— replacing X% of traditional lecturing by a 
more suitable method; 


— starting a bank of examination questions 
that meet the criteria of objectivity, 
validity, etc. (X questions); 

— calculating the discrimination index or 
acceptable level of performance for X% 
of the examination questions; 


oi & 
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how to plan a mini-workshop 


You have already had first-hand experience, 
as a participant, of an educational workshop 
and you have decided to organize and run a 
mini-workshop yourself in order to let your 
colleagues know about the systematic 
approach to educational problems. This will 
involve a great deal of work. The following 
checklist has been drawn up to help you. 


Only those items considered essential appear 
on this list. Not all of them may be applic- 
able in your case and you may also find that 
some items you need are missing. 


Unless you start work at Jeast six months 
beforehand you will be increasing your 
chances of failure. 


The greater the flexibility and adaptability 
of equipment and staff the greater the 
chances of success. 


One of the aims of the workshop is to meet 
the needs of the participants. Your apologies 
will be no use to them if something goes 
wrong. On the other hand, they will be 
favourably impressed if, when something 
unforeseen does happen, corrective measures 
are taken to keep the workshop running 
smoothly. 


No matter what you do the unexpected will 
always happen! 
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A) “etcetera 2 Oa a) RD 


It is a tragedy that as soon as normally responsible 
adults come into contact with education they expect 
to be told what to do and what to learn... Teachers 
play along with this and find it much easier to meet 
these expectations than to create the conditions in 
which students will take responsibility for their own 
learning. 


B J Ed. Tech. Jan. 76 N. Farnes 
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The success of a workshop will depend largely on the 
Way it is planned and on the arrangements made before 
the opening session. 
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action checklist 


edieershy see Action to be taken 
days | page 


— Decision to organize a mini-workshop 


— Open a file 
— Define the general objectives and aims of the 
workshop 
=~ 365 — Find a source of funds 
to — 180 — Have a draft budget approved 


— Set the dates for the workshop 

— Choose the place to hold the workshop 

— Book a meeting room and accommodation 
for the participants 

— Define the criteria for selecting participants 

— Appoint a Committee of Sponsors 

— Choose the assistant organizers 

— Take account of the working language 


| 

" Start the procedure for inviting participants, 
$e informing them of the aims ig i workshop 
| and sending them doc. 1 (Working methods) 
and doc. 2 (Theme of the workshop) 


mmOninth 


— 60 ~ Select the participants from those applying 
— 45 — Inform participants that they have been 
selected, and 
= 30 send them the documentation (Educational 
Handbook) 
— Arrange for document reproduction 
equipment to be available 
— Prepare a checklist of the equipment required 
— Inform the press 


| —8 — Review the list of participants 
| 7: — Arrange the room and inspect the premises 
(with equipment checklist) 
| — Call a meeting of the assistant organizers and 
review the programme for the workshop 
—1 — Have a friendly drink 


Background organization of the workshop ... 
— timetable of work 
0 — functioning of the workshop 
— organizing the breaks 
— group photograph 
— immediate evaluation (doc.3) 


AZZ 


ee ee ee 
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+ 30 — Send the report to the participants wh 
to the responsible authorities 


+ 180 — Start long-term evaluation 
to 365 | — collect data 
) 


hn > 


— yisit the participants | 
— organize an evaluation meeting 
— publish an evaluation report 


Open a file 


Correspondence relating to the planning, 
running and evaluation of a workshop will 
soon reach proportions that call for proper 
filing. A suitable system might be a loose- 
leaf file with the following subdivisions: 


— Budget 

— Workshop site 

— Selection of participants 

— Selection of assistant organizers 
— Documentation 

— Equipment checklist 

— Publicity, press, etc. 

— Evaluation 


Aims of the workshop 


In the particular case of an educational work- 
shop, the aims have already been described 
on page 5.03. If the workshop has different 
aims, it is then necessary to define them 
explicitly. 


Funds 


Whatever social and political system you are 
working under you will need a budget. The 
person or persons who will ultimately be 
responsible for authorizing the expenditure 
involved will need at least one estimate. 
To work this out the following simple 
formula is suggested: 


B=(7+S)N x 1.25 


E= Estimate 


T= Costs of return travel* 
S = Living expenses ding 
(accommodation, food) participant 


N= Number of participants 


In other words, travel costs and living 
expenses will amount to 80% of total costs, 
leaving the remaining 20% to cover the other 
expenses (room, reproduction of documents, 
etc). Where necesary add a percentage corres- 
ponding to the annual rate of inflation 
and get your budget approved. 


Date of the workshop 


As the workshop will nearly always be 
attended by teachers, this should be taken 
into account when setting the dates of the 


* Before working this out, read the paragraph on “choosing 
the place for the workshop” on this page. 


5.06 


workshop in order to avoid clashing with 
their professional commitments (¢€.g. sitting 
on an examinations board, annual congress). 


You should also check whether the dates 
coincide with public or religious holidays, 
sports events or political meetings, as these 
may create problems with regard to reserving 
hotel rooms. 


It is recommended that the first day of the 
meeting should not immediately follow a 
non-working day (such as Sunday, or Friday 
in Moslem countries), so as to ensure that at 
least one working day will precede the 
opening of the workshop. 


Choosing the place 
for the workshop 
(over 6 months before D-day) 


It has been found from experience that, to 
make sure participants will attend on a 
full-time basis, the workshop will have to be 
held in a place far enough away from where 
participants live to enable them to take part 
in all activities without interruption and 
prevent them from being able to go home 
after the sessions or, more importantly, 
return to their laboratories or their patients. 


This obviously implies a substantial financial 
investment but one that is justifiable from the 
point of view of cost/effectiveness. 


The place chosen should preferably be secluded 
but agreeable and the conditions comfortable 
enough for participants to be able to recall 
with pleasure thei first full-time plunge 
into the depths of the systematic approach to 
education. 


Booking a meeting room 
and hotel accommodation 


(6 months before D-day) 


Waste no time in making the necessary 
bookings. Do it in writing and insist on 
written confirmation detailing the conditions 
you have specified, particularly as regards 
the meeting room (see p. 5.17 for details). 


Ensure that the meeting room will be avail- 
able 24 hours a day throughout the work- 
shop. Wherever possible, a visit to the 


premises is recommended before making a 
final decision. If the meeting room is too 
small, too noisy, badly ventilated or poorly 
lit, the workshop may suffer irreparably. 


Criteria for selection 
of participants 


Number of participants 


It has been found from experience that as 
many as 35 participants (seven groups of five) 
can be handled by one organizer. By 
following the maxim “‘the less you teach the 
more they learn’, it should be possible to 
increase the number of participants still 
further. However, there are no strict rules 
and it is advisable not to have more than 
about 15 participants for a first trial. 


The number of participants is in practice 
limited by what the organizer feels he is 
capable of coping with and by what the 
participants think they need in the way 
of help from the organizer. 


The documentation provided for the mini- 
workshop should enable each participant to 
progress by his own efforts and with the help 
of the stimulation provided by _ other 
participants rather than with the help of the 
organizer. 


Type of participant 
* Homogeneity 


Although it is not necessary to have all the 
same kind of participants for a mini-workshop, 
since this type of workshop is mainly 
intended to provide a stimulus to the 
individual, it is advisable to select participants 
who will be in a position to work together 
after the workshop is over and who, by 
forming a critical mass, will have a greater 
chance of success. 


On the other hand, in the light of experience 
it is not recommended that a workshop 
designed as an introduction to a new 
approach should be attended by a mixed 
group representing several professions (for 
example: physicians, nurses, dentists, medical 
assistants, sanitary engineers, etc.). The 
educational mini-workshop is going to force 
each participant to question many concepts 
that had previously seemed firmly established. 


This process is hard enough without increasing 
frustration by insisting that it is carried out 
outside the peer group. 


The aim at this stage is not to create team 
spirit. It may be necessary to organize a 
mixed group later on in order to reach this 
goal. 


* Voluntary participation 


and willingness to innovate 


Attendance at the workshop should be 
voluntary and each participant should already 
have demonstrated his desire for change by 
having adopted new methods of his own. He 
should preferably be in a position of res- 
ponsibility or be likely to acquire respons- 
ibility. 


In cases where some of the documentation is 
not yet available in the national language, or 
if assistant organizers who do not speak the 
national language have to be called on, 
allowance must be made for this and 
participants selected who can at least read the 
language used in the documents available. 


Please Note! 


To be accepted as a participant it ts 
absolutely essential that: applicants are 
aware they will be full-time participants 
for the duration of the workshop and 
undertake to abide by this provision. 


Important! 


Committee of sponsors 


Setting up a Committee of Sponsors will not 
only give you an opportunity of honouring 
influential members of official circles but 
will also draw the attention of such circles 
to the action you have initiated and 
encourage them to follow it up. It is 
important that people in administrative 
positions (such as Rectors, Directors of 
Health and Deans) should be represented on 
such committees, which will be called on to 
apply the selection criteria defined earlier. 


Selection of assistant organizers 


(4 months before D-day) 


If you have already attended a workshop of 
this kind yourself, do not be afraid to take 
sole charge of a group of about 15 participants. 
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However, if you do not have enough 
confidence in yourself yet, call in a more 
experienced consultant, preferably from 
another school or faculty, and this will allow 
you to take a rather larger group (20-25). 


In subsequent meetings with larger numbers 
of participants you are strongly advised to 
take on two (for 20 participants) or three 
(for 30 participants) assistant organizers 
(from those attending an earlier workshop). 
You will have to make sure at least four 
months before the workshop starts that they 
will be able to come, so as to give them time 
to make further study of the documents, in 
particular the Educational Handbook. 


You are advised to choose assistant organizers 
belonging to disciplines other than those 
represented by the participants, to prevent 
the proceedings turning into a discussion of 
their subject by experts rather than a con- 
sideration of the methodology of education. 


The assistant organizers will have the task of 
finding answers to questions put by the 
participants and of channelling any questions 
that they cannot deal with themselves to the 
principal organizer. 

Working language 

Apart from the WHO Educational Handbook,’ 
the remaining documents (1, 2 and 3) are 
short enough to be translated into the 
language of the participants of the workshop. 
At any rate all discussions, whether in small 
groups or in plenary session (unless outside 
consultants are used), may obviously be 
carried out in the national language. Make 


arrangements for any translations to be done 
at once. 


Invitation to the participants 

(4 months before D-day) 

It is now time to start the procedure leading 
up to the final selection of participants 


(see page 5.15 ). Where possible a demand for 
places in excess of the number you have 


'Which has been translated into Arabic, Bulgarian, Czech, 
French, German, Hungarian, Indonesian, Italian, Persian, 
Polish, Portuguese, Russian, Serbo-Croat and Spanish. 
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decided on (page 5.07) should be created 
so that you will be able to correct the 
situation if there are any last minute can- 
cellations. You are therefore advised to get 
in touch right away with the colleagues you 
wish to contact. A personal letter will 
generally be preferable to posting up a notice, 
but your decision here will depend on local 
customs. What “information” should be sent 
out at this stage? The main points to be 
covered are: 


(a) aims of the workshop — prepare a letter 
of invitation based on the content of 
p. 5.03 as adapted to the target popul- 
ation. 


(b) What is implied by the term workshop 
—you may annex to your letter the text 
on page 5.03. 


(c) working methods of the workshop 


(document No. 1) 


(d) theme of the workshop (document No. 
2) 


The text of the last two documents mentioned 
(1 and 2) will be found on the pages that 
follow. The required number of copies can 
be made by means of a stencil if you do not 
have access to other methods of reproduction 
such as an electronic stencil or a fast photo- 
copying machine, for which the pages of this 
document may be used as originals. In any 
case the pages should be renumbered and the 
place and dates of the workshop should be 
mentioned on the cover page of each docu- 
ment under the title “Workshop in educational 
planning’. 


Your letter of invitation should also mention 
that full-time participation is essential (see 
box, page 5.07) and indicate any language 
stipulations. 


Lastly, you should set a deadline for applic- 
ations (45 days after the date of dispatch of 
the letter of invitation), mentioning that each 
successful applicant will be informed of his 
selection at the latest 45 days before the 
start of the workshop and that the basic 
documents will be sent to him at that time 
(under separate cover if you have a limited 
budget). 


workshop in educational planning 


document 7 


working methods of the workshop 
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working methods 


The working methods proposed for this 
meeting may be somewhat different from 
those you are used to. It does not mean that 
these are new methods: they have been widely 
used outside the university and their effective- 
ness has been experimentally tested. They 
derive from the application of recent education 
theories. If they were to be identified by their 
main characteristic one would say they put 
emphasis on active participation of the learner 
while in traditional systems he is maintained 
in a relatively more passive role. The aim of 
the methods is to encourage you to develop a 
critical and constructive attitude and to find 
solutions for your own problems. Practical 
exercises raising specific problems that 
simulate real situations will lead you to pro- 
pose valid solutions. Generally speaking, the 
working methods used in the workshop will 
enable you to put the educational principles 
recommended in the WHO Educational Hand- 
book into practice. 


1. Free choice of personal objectives 


To ensure that the workshop fully meets 
your educational needs you will be invited to 
select the objectives you wish to reach by the 
end of the workshop (see p. 5.19). The 
meeting’s programme of work will be 
organized on the basis of the choices made 
cpr >.20). 


2. Preliminary reading assignments 
(concerning objectives chosen). 


To provide you with the theoretical know- 
ledge in the field of educational planning that 
you may need to find solutions to practical 
problems and attain the objectives you have 
chosen, you will be invited to read certain 
documents (see p. 14). It should be made 
clear at this point that there will be no 
lecturing on the part of the organizers, not 
even a “‘short introduction to ...”. Study of 
the recommended texts will be your own 
responsibility. 


3. Clarifying sessions 


These are generally held as the first working 
session of each day. Their aim is to ensure 
that participants have a clear idea of what 
they are to do in the hours that follow and 
that any instructions have been understood. 


5.10 


They are not intended, however, to be 
in-depth discussions, which will be held either 
during working groups or during summing-up 
sessions. 


4. Practical exercises 


These exercises, which are described in detail 
in the Educational Handbook, are to be done 
individually at first. Then discussion will 
start by comparing each participant’s pro- 
posed solutions. This exchange of views may 
take place in pairs before extending to the 
entire small group; if the exercise implies that 
each one within the group has a specific task 
related to a common goal, it is up to the 
group to get organized. This task distribution 
will mot be done by the organizers. Group 
dynamics will operate with its highs and lows, 
periods of tension and relaxation as in all 
human endeavours. 


5. Group presentation 


Plenary sessions will be held as often as 
necessary, depending on the programme of 
work based on the personal objectives chosen 
by the participants. Their aim will be to 
allow group solutions to be presented, not 
for the purpose of judging groups but to com- 
pare ideas for possible consensus under the 
leadership of a participant or, exceptionally, 
an organizer. 


6. Preview of next working day 


Each day before closing a short period will be 
reserved for a preview of the following day’s 
activities and a reminder of the aims of 
reading assignments. 


Any changes of programme will be called to 
the participants’ attention. 


7. Individual consultations 


In view of the biological principle of individual 
differences it is most likely that participants 
will progress at different rates, desiring to 
study in more or less depth certain questions, 
or be interested in differing applications of 
the theories and methods proposed during the 
workshop. The organizers will therefore make 
themselves available on request for individual 
consultations on subjects of special interest. 


8. Formative evaluation 


8.1 Pre-test - To help direct your efforts 
and inform you of your own progress during 
the workshop, an evaluation process is 
proposed. Details are given on page 9 of the 
Educational Handbook. This test will help 
you to identify the ‘‘educational areas’’ which 
may need attention. The object of the pre-test 
is not to find the “right” answer but simply 
to register that you have not found it. The 
purpose of the workshop is to help you to 
find it, either during the workshop itself or 
in the ensuing months. The post-test will 
enable you to measure your progress as time 
goes by. 


8.2 Daily personal evaluation — You should 
assess your own progress each day. This will 
give you an opportunity of modifying the 
objectives you chose (para. 1) on the first 
day. 


8.3 Daily group evaluation — At the end of 
each daily session time will be set aside for 
joint assessment of the extent to which the 


working methods used have helped you 
towards reaching the objectives you have 
selected, and for proposing any change in 
the functioning of the workshop. 


8.4 Evaluation questionnaire — Shortly before 
the end of the meeting you will be invited 
to express your opinion on the organiz- 
ation of the workshop by means of a 
questionnaire (document 3, p. 5.25). The 
results will be analysed during the last session. 


8.5 Long-term evaluation — To help you 
assess the benefits you will continue to reap 
from this workshop, you will be asked to 
define explicitly the professional objectives 
you expect to reach within the next twelve 
months in relation to the experience acquired 
during the workshop. It is recommended 
that all participants meet twelve months after 
the workshop to assess what they have 
achieved. 


These various aspects of the working methods 
that will be used during the workshop will 
be clarified during the first session. 


Julie 


You may now start preparing for the workshop in 
whatever time you have available. 


UA 


— If you want to get a general idea of the field that will 
be explored during the workshop, study document 
No. 2 (Introduction to Educational Planning). 


nO 


— If you want to select the personal objectives you 
would like to reach by the end of the workshop, refer 
to pages 11 —13 (Identification of your needs as an 
educator) of the Educational Handbook, which you 
will be sent if you are selected to participate. 


, 


— If you want to go still further, you can use the 
Educational Handbook to pursue the objectives you 
have selected. 


Good Lush’ 


workshop in educational planning 


document 2 


theme of the workshop = 
introduction to educational planning 


Note to organizer of educational workshop 


It is suggested that in preparing this document you use the text that appears on pp. 1.05 — 1.07 
of the Handbook or the section on ‘Distinctive attributes of education for the professions”’ 
(pp. 3.05 — 3.12). 
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The theme of the workshop will therefore be the health 
manpower training process, covering the four main 
stages (see the Educational Spiral, p.1.06): 

— definition of relevant educational objectives 

— planning of an evaluation system 

— development of an effective educational programme 


— application of a valid system of evaluation 


Selection of participants 
(2 months before D-day) 


The deadline for applications has now expired 
and you should convene the Committee of 
Sponsors. They will choose from among 
the applicants those who correspond most 
closely to the criteria defined earlier (see 
page 5.07). Your function will be to make 
sure that the Committee follows these rules 


properly. 


In addition to the number of participants 
decided on, the Committee should select 
some reserves (20—25% extra) to provide 
replacements in the case of last-minute 
cancellations. 


Confirmation of participants 
(45 days before D-day) 


It is now time to write to the applicants who 
have been selected, reminding them of the 
conditions of participation (full-time 
attendance compulsory) and of the place 
and dates and sending them the Educational 
Handbook, with a recommendation that they 
reread p. 5.12 if they wish to start work. 


In point of fact, distribution of the Handbook 
could just as well wait until the start of the 
workshop since it is not essential to study it 
beforehand. However, it has been found from 
experience that many participants complain 
at the time of final evaluation that they were 
not given al] the documentation before the 


workshop (including those who would not 
have had time to read it). In short, although 
it is not essential for participants to receive 
the Handbook one month before the work- 
shop, there is no reason why they should 
not have it and there may be some advantages. 


Staff and equipment needed 
for document reproduction 


during the workshop 
(One month before D-day) 


It will be extremely useful to have a secre- 
tary or typist available during the meeting for 
typing the documents resulting from group 
work. Since participants generally wish to 
have access to the results of their colleagues’ 
work, equipment for fast, good quality 
reproduction will also be needed. Each docu- 
ment should have a reference number for ease 
of consultation. 


The secretary can also help with logistic 
matters (hotel rooms, problems concerning 
transport, finance, etc.) on the participants’ 
arrival, thus freeing the organizers to spend 
more time on purely educational activities. 


Now is also a good time to make copies of 
page 5.20 , which each participant will need 
on the first day, and of document 3, 
“Evaluation of the workshop by _ the 
participants‘, which you will distribute 
towards the end of the workshop (see p.5.25). 
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Select participants who are most likely to benefit from 
the workshop. Be consistent in maintaining contact 
with them. 

Carole J. Bland 
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Note 

Now is the time to order however many copies you need 
of the WHO Educational Handbook for Health Personnel, 
unless your national authorities already have a stock 
that you can use. 
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Equipment checklist 
(One month before D-day) 


Note pads (one for each participant + 20%) 
Pencils (one for each participant + 50%) 
Rubbers/erasers (one for each table) 

Pencil sharpeners (one for each table) 
Two-hole punch (1) 

Adhesive tape (2 rolls, incl. one wide) 
Stapler (1) 

Waste paper baskets (one for each table) 


Projection screens (2) or white wall 


Cardboard envelope files (2 for each participant) 


Overhead projectors (2) 

Spare projector lamps (2) 

Electric extension flex (6 metres) 

Electric adapter plugs (2) 

110/220 V transformer (check local voltage) 
Transparent cellulose sheets (50) 


Marker crayons for writing on cellulose sheets 
(water soluble) (12) 


Blackboard or, preferably, large flip-chart 
Photocopying machine 
Dictionary 


Extra copies of Educational Handbook (20%) 


You will find this list useful for checking what you should bring with you to the workshop and for 
making sure the day before the workshop begins that everything is in place. 


Press relations 


Depending on the local situation, it may be 
worth deciding to inform the press. If so, it 
is always best to prepare a press release your- 
self rather than leaving this task to a journalist, 
no matter how conscientious he may be. 


If you invite the press to interview the parti- 
cipants (for example when the group photo- 
graph is being taken) the best time for this 
will be during the break (see p. 5.24) on the 
last day (or the last day but one), This is also 
a good time to invite your superiors, those 
providing funds and other dignitaries. Never 
choose the first day for this, as the 
participants are likely to be in a state of 
considerable confusion! 


~ 


Review of the list of participants 
(one week before D-day) 


In some cases there will be cancellations. 
As these occur you should get in touch with 
the applicants selected as reserves to fill the 
empty places. Make sure once again that 
participants have all the documentation 
required and that they know the date and 
place of the workshop. 
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Check regularly what you have done and what 
remains to be done. 
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Arrangement of the room 
(2 days before D-day) 


spare chairs 


35 participants 
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25 participants 


15 participants 


or) 


6 


(depending on the number of participants) 


The meeting room should be arranged so as to: The arrangement shown above is suggested 


— allow participants to sit at small tables in for 15, 25 or 35 participants. 


groups of three to five; During plenary sessions, participants just have 


— allow the use of an overhead projector to turn to face the organizer. Make sure that 
(two would be preferable so that two everyone has a good view of the projection 
documents can be compared). screen and the discussion leader. 
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It has been found from experience that the 
noise threshold (in group discussions) is quite 
bearable and that this arrangement 1s pre- 
ferable to separating groups in different 
rooms. It allows for much more flexibility 
in organizing the sessions. 


Make sure that the room is not too near a 
source of noise (restaurant, school, 


demolition site, etc.). 


The less the meeting room looks like a “class- 
room” the better. 


Make sure that each participant has enough 
table space to lay out his documents and 


that it is well lit. 


The overhead projector does not require a 
darkened room, but you should still make 
sure the day before the workshop starts, 
at a time when natural daylight is at its 
brightest, that the picture projected is clearly 
visible. Make sure that there is at least one 
electric power point that works and have an 
electric extension flex and spare projector 
lamp available. In addition have a stock of 
transparent cellulose sheets and marker 
crayons (erasable) available so that 
participants, or you yourself, may illustrate 
any remarks that may be made in plenary 
sessions. 


Use the checklist of the equipment you will 
need during the workshop when making your 
last inspection of the room the day before 
the meeting starts (page 5.16). 


Coordination of 


assistant organizers 
(2 days before D-day) 


All assistant organizers should be on the spot 
without fail at least two whole days before 
the start of the workshop and should have 
been told how important this coordination 
period is. 


The principal organizer should make sure 
that each assistant organizer knows what is 
expected of him during the workshop and is 
prepared to carry it out. 


The two days preceding the workshop will 
therefore be a sort of dress rehearsal for what 
Is expected to happen during the workshop. 
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Each assistant organizer ought to give an 
account in his own words of the part he 
thinks he can play. 


All organizers will need to be thoroughly 
familiar with the documentation for the 
workshop and these two days provide a good 
opportunity for organizers to exchange views 
and prepare the way for working together 
smoothly. Several informal meetings will 
probably be useful, one of which should be 
in the room in which the workshop is to be 
held. 


The organizers must also decide what criteria 
to apply (mixing of disciplines, grades, sexes, 
natural leaders, “heavies’’, etc.) in dividing 
the participants into groups (of three to five) 
and assigning them places, which should be 
marked by name-cards. 


The evening before 


In theory all the participants will have arrived 
and have their hotel rooms. Before supper, 
it is recommended (if finances allow) that you 
organize a “‘friendly drink” to break the ice 
and enable participants and assistant 
organizers to make themselves known to each 
other. This should be as informal as possible. 


D-day is here.... 

Although it is in the nature of a workshop not 
to have a strict hour-by-hour timetable, in the 
case of a mini-workshop some details may be 
given on the procedure it is recommended to 
follow, at least for the first day. The first 
hours of the first day are of crucial 
importance. The working atmosphere will 
change in the course of the three days: 
roughly speaking the first day will be one of 
confusion, the second one of productive 
thought and the third one of stunned realiz- 
ation that there is much more to learn than 
appears at first but that it is worth making 
the attempt and that this is only the 
beginning... 


With regard to the rate at which the subject 
matter is dealt with in the time available, 
each participant should be left to work at his 
own pace and according to his own system of 
priorities. On the other hand, as_ the 
participants form themselves into working 
groups, a “common tempo” will be 
established. 


programme preparation 


In order to prepare a working programme for the workshop that is 
relevant to your own needs in the field of education, you are invited 
to choose from among the objectives listed on pages 12 and 13 (divided 
into four main themes) those that interest you and that you would 
like to achieve by the end of the workshop. 


As the duration of this workshop is limited, try to be realistic in your 
choice. Some of these objectives may require only a few minutes’ 
work; others several hours to allow for study of the documents made 
available to you (see suggested texts, p. 14). 


To make it easier to choose, rearrange the objectives listed in order of 
importance to you. Once you have made your choice, fill in page 5.20 
and hand it to the workshop organizer. 


In the light of what you have selected, a programme of work can be 
drawn up. You will be given a list of the participants who have chosen 
the same objectives as yourself, so as to facilitate the organization of 
small working groups engaged in common activities* (see example, 
p. 5.21). 


It is natural that you should have some difficulty in making your choice 
at this early stage in the meeting. Do not hesitate to consult an 
organizer . . . and above all remember that if necessary you can always 
modify your choice during the workshop. 


*Some of the objectives may be achieved more easily by group work. These objectives are marked on pp. 12 
and 13 and p. 5.20 by an asterisk”. 
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Circle the number corresponding to each objective you have chosen 


Additional Objectives 
(optional) 


By the end of the workshop I should like to be able to: 


Name cis .ccondiew. 26 bic een ebetam & 
[block capitals] 


Hand a copy of this page to the organizer of the workshop before the break in the first session. 
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How to prepare a working programme based on the objectives selected by the participants 


List of participants according to choice of 
objectives suitable for group work (no more 
than five per group) 


Proposed programme of work 


3 Brian, Alan, Ian, Patrick, Alfred, Claudia, 
4 Elise, Emily, Margaret, Fred, Michael, 
Alfred, Albert, Julian, Harriet, Frank, 
: 
5 Claudia, Emily, Alan, Frank, Brian, 
Margaret, Patrick, Andrew, Philip, Elise, 
to Ian, Jack, Harriet, Alfred, Alan, 
8 
16 
Michael, Alan, Emily, Patrick, Claudia, Ian, | Group work 9 — 10.30 am 
24 Jack, Harriet, Alfed, Philip, Brian, Plenary Session 11 am — 12.30 pm 
27 
28 
9 
18 
29 


Group work 3 — 4.30 p.m. 


Plenary Session 5 — 6 pm 


Tuesday 


Group work 9 — 10.30 a.m. 
Plenary Session 11 am — 12.30 pm 


Wednesday 


Group work 3 — 4.30 pm 


Plenary Session 5 — 6.30 pm 
Wednesday 


Felix, Jack 
Julian, Michael, Felix, Fred 
i Emily, Alfred, Frank, Harriet, Ian, 
Elise, Frank (3) Thursday 


Andrew, Jack, Julian, Fred, Michael, 
Felix, Claudia, Brian, Alan, Patrick, Elise 
Alan, Michael, Margaret, Julian, 
Andrew, Felix (6) Individual consultations 
Thursday 


(4) Individual consultations 


32 Peter, Fred, Felix, Margaret, Elise @ | Individual consultations 


Felix, Michael, Julian, Alan, Emily, 
Alfred, Harriet, Jack, Ian, Patrick, 
Elise, Andrew, Margaret, Brian, Frank, 
Philip, Claudia, Albert, Peter 


Group work 3 — 4 pm 


Plenary session 4— 4.30pm 
and 5.30 — 6.30 pm 


Felix, Michael, Julian, Alan, Emily, 
Alfred, Harriet, Jack, Ian, Patrick, 


Elise, Andrew, Margaret, Brian, Frank, 
Philip, Claudia, Albert, Peter 


4 


Felix, Margaret, Peter, Fred 


33 to 35 | Elise, Frank, Andrew, Margaret, Fred G) Individual consultations 


Fred, Peter, Albert, Julian, Felix (s) Individual consultations 
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Example of a proposed programme prepared on the basis of 
the objectives selected by participants in a workshop 


The length of the arrows indicates the percentage of participants who have chosen any one 
objective. The actual number is circled. 


First day: first session 


8 to 10.30 am 


1. Opening 

In your opening remarks you will of course 
formally welcome the participants, thank 
those who have made the workshop possible, 
recall the overall aims of the workshop 
(see page 5.03) and relate it to the local 
teacher training situation. Make clear too that 
the workshop offers each participant a golden 
opportunity for uninterrupted thought on 
problems that are universally admitted to be 
important but are frequently neglected; that 
no-one there is any cleverer than anyone else; 
that the workshop belongs to the participants 
and will be what they make of it; and that the 
third day will not be the end of the workshop 
but rather the start of a long and exciting 
process. These remarks should not take more 
than five minutes. 


2. Clarification of documents 


Go on without a break to this item. Ask the 
participants to turn to document No. 1 
(Working Methods) and go through it page by 
page and paragraph by paragraph asking them 
if there are any points they would like 
clarified. If there are no questions on a 
paragraph, describe its central theme without 
going into details. If a question is raised, ask 
whether any other participant would like to 
clarify the point concerned. Do not forget 
(and remind participants where necessary) 
that the object of the exercise is to clarify 
obscure points and not to discuss the subject 
matter in depth. There will be time for such 
discussion throughout the rest of the 
workshop either in small working groups 
or, occasionally, in plenary session. 


It has been found from experience that there 
is no point at this stage in a clarification 
session for document No. 2 (Theme of the 
workshop). The necessary definitions and 
detailed explanations are dealt with in the 
exercises designed to help achieve the work- 
shop objectives. 


It will now be between 8.45 and 9.15 am and 
time to pass on to the next item. 


ay 


3. Programme preparation 


The (individual) programme of work for each 
participant will now be prepared. Ask 
participants to turn to page 5.19 of the 
Handbook (Programme preparation) and 
make sure that the text of the first page has 
been understood. Next, mention that the 40 
objectives listed are the same as those in the 
Educational Handbook and that the cross- 
references to the relevant pages of the Hand- 
book given on page 14 are intended to 
facilitate selection of objectives. The selection 
made should be indicated by filling in page 
5.20 and handing it in to the organizers by 
10.30 am at the latest. During the break, 
using the information produced in this way, 
you will draw up a list of participants who 
have chosen identical objectives so that they 
may form themselves into groups for joint 
work (see example p. 5.21). 


The first plenary session will now be over and 
the time will be between 9 and 9.30 am. 
From this point until the start of the break 
(10.30 to 11.00) it will be the task of the 
assistant organizers to deal individually with 
any requests from participants. Towards 
10.00 am make a rapid tour of the room to 
see how the participants are doing and by 
about 10.15 remind them if necessary that 
they have 15 minutes left for handing in page 
5320: 


First break 
(10.30 to 11.00 am) 


This is when the organizer will draw up the 
list (on a flip chart) of the participants who 
have selected identical objectives so that they 
can form groups for joint discussion. The list 
will enable the organizer to prepare a draft 
programme of work for the rest of the work- 
shop. Any objective that has attracted the 
interest of one-third of the participants 
should be dealt with in plenary session. Since 
each presentation will require preparatory 
work by groups of participants (no more than 
five per group), enough time should be 
allowed for this (either during morning 
sessions or in extra sessions held in the after- 
noons or, exceptionally, the evenings). Since 
you have the gift of being everywhere at once, 


you should also make sure that coffee, tea or 
other drinks, and rolls and sandwiches are 
available to the participants. 


First day: second session 
11.00 am to 1.00 pm 


At about 10.55 am invite the participants to 
come back to the room. Present the draft 
programme (see example, p. 5.21) and as 
soon as it has been accepted tell the 
participants that they can start working (on 
an individual basis at first) through the list 
of objectives they have selected; that if they 
wish to find out what they know, they 
should now take the pre-test (at least on the 
first chapter of the Handbook); that they 
may call on you if they need any help; that 
they may subsequently, if they wish, start 
exchanging their work and discussing it with 
other participants, whom they may choose 
from the list given on the flip chart (which 
you may now have reproduced for general 
distribution). 


Until then take no other action apart from 
being available to give help if necessary. 


At 12.55 pm you will have a few minutes 
left to congratulate participants on their 
dedication and keenness and reassure them 
if they are feeling completely confused about 
what is happening that things will be better 
tomorrow and even better the day after 
tomorrow. Lastly, invite them to spend the 
afternoon continuing to clarify their ideas 
by further reading on any points worthy of 
attention and tell them that you will be 
available for individual consultation. 


It will now be 1.00 pm. Wish everyone a 
pleasant lunch and confirm the time and the 
aims of the next activity on the programme 
that has been approved.’ 


First day: afternoon session 

Begin by reminding participants of the time 
of the next plenary session. Make sure that 
each small group knows what it is supposed to 
be doing, then give no further help unless 
asked. You can, however, go around the room 
taking a friendly interest in what each person 
is doing: adapt your attitude to the reactions 
shown by the participants. See that each 
group has a supply of transparent cellulose 
sheets and marker crayons. 


Ten minutes before the time set for the 
session, check that each group is almost 
ready. As for the general organization of the 
plenary session, all you need to do is to 
present the transparencies prepared by each 
group, using the overhead projector, and 
encourage discussion. 


Fifteen minutes before the end of the last 
session of the day, call for everyone’s 
attention. Invite the members of each group 
to discuss among themselves the positive and 
negative aspects of this first day and ask for 
a verbal report from a spokesman for each 
table. 


Second (and third) day 


The participants will start organizing their 
work themselves and your task will be to help 
them do this. Start the day with a short 
clarification session (15 minutes) which this 
time it will be worth your while to initiate 
within the groups and finish in plenary 
session. Next, mention that as on the previous 
day there will be a plenary session at the end 
of the day for evaluation of the day’s work 
and invite everyone to set to work. Add that 
in the event of one (or several) working 
groups forming and wishing to submit the 
results of their work to the others, further 
short plenary sessions may be held on request. 
Make sure that, should this happen, there are 
enough tranparent cellulose sheets and marker 
crayons for participants to make their reports 
using the overhead projector. 


At the end of the session, after evaluation of 
the day’s work, invite participants to draw up 
for the following day a list of the professional 
objectives they wish to reach during the next 
12 months in the light of what they have 
‘“Jearned”’ during the workshop, and ask them 
to fill in the evaluation questionnaire 
(document No. 3) and return it to you by 
8.30 next morning at the latest (page 5.31 will 
tell you how to analyse the results of the 
questionnaire). 


1 Another scenario may be imagined in which by 8.30 am 


the participants have settled down to a poker game. This 
would indicate that you need to review your planning 
procedures, including the method of selecting participants 
(page 5.07). 
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Last day 

(Don’t forget to have a group photograph 
taken during the break, unless it has already 
been done). 

After a short period for clarification, remind 
the participants that it is very important to 
devote most of this last day to finalizing the 
individual professional objectives to be 
reached during the next 12 months. (See 
document No. 1, paragraph 8.5). These 
should be the subject of exchanges of views 
between participants and of group discussion 
before being handed to the organizers. The 
last half-hour of the last day should be given 
over to an evaluation session (in the light of 
the analysis of the questionnaire/document 
No. 3), ending with a few closing remarks 
dealing mainly with the future. 


Letters of thanks 
(15 days after the workshop) 


It is now time to thank everybody who has 
helped you, including those who have 
provided funds for the workshop, the members 
of the Committee of Sponsors and the 
assistant organizers, if any. Inform them in 
the letter that they will shortly be sent the 
report on the workshop. 


Report on the workshop 
(15 days after D-day) 


Even if the purpose is only to inform those 
who have provided the funds for the work- 
shop or to help participants to inform other 
colleagues, it is a useful exercise to prepare 
a report during the weeks following the work- 
shop. 


5.24 


The report will contain: 


— an introduction giving a brief description 
of the local context which led to the 
organization of the workshop 


— the general aims of the workshop 


— a description of the operation of the 
workshop mentioning how participants 
adapted to the working methods 


— some selected samples of the results of 
individual or group work 


— the list of documents used during the 
workshop 


the results of immediate evaluation, 
including an analysis of the questionnaire 
(document No. 3) 


— the list of participants (with their 
addresses) 


Long-term evaluation 
(at the latest one year after D-day) 


Long-term evaluation is essential since it is the 
only way to measure the actual impact of 
a workshop. 


It will be based on an assessment of the 
extent to which each participant has reached, 
failed to reach or exceeded all or some of 
the professional objectives he set himself at 
the end of the workshop (see page 5.23). 
An assessment of this kind may be made by 
means of questionnaires, by individual inter- 
views or, preferably, by bringing all 
participants together again for a one-day 
meeting. A report summing up the results of 
long-term evaluation is well worth preparing 
and distributing. 


workshop in educational planning 


document 


evaluation of the workshop 
by the participants 


Instructions for questions 1 — 35 


Use the following code to indicate the extent to which you agree or disagree with each of 


the statements made below: 


Code 

1 Strongly disagree 
2 Disagree 

4 Agree 

5 


Agree strongly 


The difference between 1 and 2 or between 4 and 5 is one of degree only. 


Example: 


If you want to express your complete disagreement with the statement, circle the figure 1 


as follows (1)2 4 5 


Please feel free to make any comments you think necessary (making reference to the number 
of the question) in the space reserved for the purpose on the last page. 
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Q.1 


Q.2 


0 Fe 


Q.4 


QS 


Q.6 


Qi7 


Q.8 


Q9 


5.26 


Aspects relating to the planning of the 
workshop 


I was given sufficient information on the aims 
and methods of the workshop before my 


arrival 


The planning of the workshop reflected the 
educational principles that were discussed 
there 


It was clearly explained to me at the start 
how I was to choose my objectives for the 
workshop 


I feel that the programme drawn up during 
the first session took my own choice of 
objectives into account 


The goals of the workshop appeared to me to 
be of immediate interest for my professional 
activities (education component) 


It was clear to me from the start of the work- 
shop that I was expected to play an active 
part in it 


Aspects relating to the relevance and utility 
of the working methods 


I found the documentation provided of an 
acceptable quality 


Enough documentation was provided to allow 
me to take an active part in the discussion of 
the subjects concerned 


The information given in the Educational 
Handbook helped me to reach the objectives 
I had chosen for the workshop 


Do not write 
in this column 


JUURUU EOE 


4 


Q.10 


Q.11 


Q.12 


Q.13 


Il 


Q.14 


Beg ie. 


Q,16 


Cel? 


Q.18 


QA9 


Q.20 


The working methods used during the work- 
shop encouraged me to take an active part 
in it 


I have had the opportunity during the work- 
shop of putting new knowledge into practice 
(exercises) 


Spending time on individual work during the 
workshop helped me to learn 


During the workshop I was given the 
Opportunity of working at my own pace 


Aspects relating to the way the workshop 
was run and to the attitude of the organizers 


The organizers displayed a satisfactory open- 
mindedness 


The general atmosphere of the workshop was 
conducive to serious work 


The organizers gave me the opportunity for 
critical comment 


The organizers made use of any critical 
comments I made during the workshop 


The organizers made every effort to help me 
reach my objectives for the workshop 


The way the workshop was conducted was in 
line with the educational principles it discussed 


The attitude of the organizers was conducive 
to “free learning”’ 
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Q.22 


@:23 


Q.24 


2:25 


Q.28 


Q.29 


Q.30 


Q.31 


Aspects relating to the organization of 
activities in the time available 


1 consider that enough time* was given for 
individual or group discussions with the 


organizers 


Enough time* was devoted to clarifying the 
documents 


Enough time* was given for discussion in 
small groups 


Enough time* was given for practical exercises 
Enough time* was given for individual work 


Enough time* was given for the presentation of 
work in plenary session 


Aspects relating to the benefits gained by 
the participants 


The workshop helped me to improve my 
knowledge of education theory 


The workshop helped me to develop a 
favourable attitude towards the systematic 
approach to educational problems 


The workshop has encouraged me to put the 


knowledge I have gained into practice after the 
workshop is over 


The workshop will help me to encourage my 


colleagues to learn and make use of new 
educational methods 


The workshop has increased my confidence 
In my ability to achieve my personal objectives 
in the medium term (within one year) 


* ; ; oe ‘ 
All questions asking for an opinion on the time spent on an activity 
must be considered in relation to the total time available for the work- 


shop. If you wish to comment on the length of the workshop as a 
whole, please doso using p. 5,30. 
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Aspects relating to evaluation of the workshop 


I felt that the pre-test and the follow-up test 
helped me to make a useful assessment of the 
knowledge I gained 


The pre-test was a useful exercise and showed 
the advantages of this technique 


The practical exercises showed the usefulness 
of “feedback” during the learning process 


I found the daily evaluation sessions useful 


aN 


as 


aN 


aN 
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With regard to the planning of the workshop, its method of work and the attitude of the 
organizers, note below and give actual examples of: 


(a) The factors which impressed you most favourably 


(b) The factors that impressed you least favourably 


(c) Total length of the workshop 


foshow | footw | 


Further comments and suggestions 


Note for the Organizer on how to analyse the answers to this Questionnaire 


A very simple analysis may ve carried out as follows. 


Take an uncompleted questionnaire and mark beside each question the answers given by each 


participant. For example, for 30 participants, the answers to question 6 might be: 


[11 

4+ 

+Ht 

+Ht 

5 

In other words, two participants considered that they did not understand from the start that 
they were expected to play an active part in the workshop while the 28 others understood this. 
Multiplying the number of answers by the corresponding coefficient gives a total of: 


Q.6 _ It was clear to me from the start of the workshop 7 sie 
that I was expected to play an active part in it 12 ae 


(2x2+10x4+18x5) = (4+40+90) = 134. 


The “satisfaction index” is calculated by multiplying this number by 20 (i.e. 100 divided by the 
maximum coefficient 5) and dividing it by the number of participants, in this case 30. This gives: 


134020. 12680 ong ay, 
30 30 


It is recommended that you then make a note of any questions with a “satisfaction index” below 
60%! . If there are none, identify the five questions with the lowest “satisfaction index”’ and then 
the five questions with the highest “satisfaction index’’. Let the participants have these results 
at the final evaluation session on the last day of the workshop. 


100 + 20 
= 60% 


| The satisfaction index is calculated in such a way that ‘‘average satisfaction” is 


VII Personal objectives 


2. Write down, for each of these 
objectives, a working timetable that 
will enable you to achieve them. 


Write down the objectives you hope 
to achieve during the year following 
this workshop, so that you can assess 
the progress made. 


Name 


Keep a copy of this page and give the original to one of the workshop organizers. 


Ide 


afterwards _._. 


Do not succumb to the illusion that everything is now going to be different in the institution 
where you work. Do not think that from one day to the next conservative elements will turn 
into reformers, passive elements into active ones or opponents into supporters. 


First, if you manage to persuade at least 20% of those taking part in the workshop to make a 
lasting and visible change in their teaching habits during the ensuing year, you may consider 
the workshop to have been a success. This will not be the case if all you do is record how many 
participants expressed satisfaction during the evaluation on the last day of the workshop (even 
if the figure is 100%). 


Secondly, your work is not yet over. Do not let the seed you have planted with so much effort 
wither away. Continue to stimulate your colleagues in responsible positions in your teaching 
institution to see that those participants who improve their teaching methods are “‘rewarded”’ 
in some way. 


Pe eA A If this is impossible, do something about it yourself — congratulate them — it will 
always be appreciated. 


A workshop 


Don’t think it will change nothing: 


Don’t believe it will change everything 
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*The questions appear on pages: 1.57 — 1.60, 2.45 — 2.47, 3.79 — 3.83 and 4.75 — 4.80. 


Recapitulative answer sheet for post test* 
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index 
and glossary 


index and glossary* 


This glossary of technical terms used in test 
and measurement, psychology and education 
is intended primarily for persons with limited 
training in test and measurement rather than 
for specialists. The terms defined are those 
most commonly found in test manuals and 
simple research reports. In the definitions, 
niceties of usage have sometimes been sacri- 
ficed for the sake of brevity, and, it is hoped, 
clarity. 


The terms and definitions included are purely 
ad hoc, for the purposes of this Handbook 
alone; they represent only the view of the 
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authors and do not necessarily have any 
validity outside the context of the Handbook. 
There is not complete uniformity among 
writers in the measurement field with respect 
to the use of certain technical terms; in cases 
of varying usage, either these variations are 
noted or the definition offered is the one 
that the writer feels is the “‘best”’. 


*Adapted from Roger T. Lennon, “Test Department”, 
Harcourt, Brace and World Inc. Revised by J. -J. Guilbert 
(WHO); G. and L. Masse, ENSP, Rennes (France); B. Pissarro, 
University of Paris (France) and J. C. Chancerel, University 
of Neuchatel (Switzerland). 


A 


Absolute criteria test 


13, 461-464 


Acceptability index 461-463, 479-480 


Acceptable level of performance 


Also called acceptable passing level (or mini- 
mum passing level) 13, 143-144, 150, 154, 
204, 221, 246, 402, 425, 429-430, 461-463, 
480 


Act 
Action performed by a person 149-152 


Act, professional 


Corresponds to a practical skill; the know- 
ledge how to do something and adoption of 
the appropriate attitude 2) 07. St). 
135-140, 143, 149, 156 


Active methods 


As opposed to the conventional methods 
described by Jean Piaget as “receptive”; an 
educational situation whereby the student 
assimilates the curriculum independently and 
progresses by means of individual exercises, 
the role of the teacher being limited to guid- 
ance of the student in choosing his/her edu- 
cational objectives and monitoring of his/her 
performance 323, 328,365,381 


Activity 

Group of acts and tasks performed by a 
person 6, 425 133102,,108,.110,137—h15, 
118-119, 123-135, 140, 156, 158, 163, 212, 
214-216, 245, 302, 307, 309-310, 313, 317, 
318-319, 324, 333, 334-336, 339, 348, 353, 
359-360, 371, 382, 404, 409, 413, 416-417, 


422-423, 526, 528 


Affective 


Concerning the feelings; affectivity is the 
combination of acts and tendencies which 
express the feelings felt towards others; it is 
situated primarily on the level of interpersonal 
relationships. It is often employed to qualify 
attitudes. The word “affect” is a neologism 
used by psychologists to designate an 
affective state. See also Attitude 246, 412 


Aim, 

The end envisaged; a teaching or educational 

goal, the result expected from a programme. 
6, 12913, 102,105. 407 goes) 148) 

133-134, 144, 159, 202, 215-219, 245, 302, 

308, 319, 363-364, 382, 405, 417, 441-442, 

461, 468, 503-511, 522 


Aptitude 


A combination of abilities and other character- 
istics, whether inborn or acquired, known or 
believed to be indicative of an individual’s 
ability to learn in some particular area. Thus, 
musical aptitude would refer broadly to that 
combination of physical and mental character- 
istics, motivational factors and conceivably 
other characteristics, which is conducive to 
acquiring proficiency in the musical field. 
Some exclude motivational factors from the 
concept of “‘aptitude’’, but the more compre- 
hensive use seems preferable. The layman may 
think of “aptitude” as referring only to some 
inborn capacity; the term is no longer so res- 
tricted in its psychological or measurement 
usage. See Skills. 


Arrangement of meeting room (for educ- 
ational workshop) 517 


Assessment. See Evaluation 


Attitude 


The internal disposition reflected by one’s 
behaviour with respect to persons, events, 
opinions or theories. In the Handbook it is 
used to denote only a relatively constant 
disposition of feeling towards someone 
(personal relations) 135-141, 155, 221, 
230, 306-312, 321, 346, 423-424, 527 


Audio-visual aids. See Media 


B 


Bedside teaching 342a 


Behaviour 
The total reactions (of an individual) access- 
ible to external observation. Thought and 


understanding are implicit aspects of be- 
haviour which are observable not directly, 
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but solely by inference from other observ- 
able types of behaviour. 13, 104, 109, 
13 aoe, 155) 157; 211=212,214, 2219233, 
246, 306-311, 318, 320, 326, 379, 402, 423 


Blackboard 344, 516 


C 


Central tendency, error of. 453 
Certifyingevaluation 12,108, 202,215-216, 
2d9, 34:5, 304, 450 


Checklist (Of action to be taken for educ- 
ational workshop). 504, 505, 506 


Checklist (Of equipment for educational 
workshop). 516-517 


Classification 


Systematic division into classes. 137-140 


Coefficient of correlation (r) 


A measure of the degree of relationship, or 
“going-togetherness’” between two sets of 
measures for the same group of individuals. 
Two coefficients of correlation are used in 
psychology, first the Pearson (r), developed 
at the beginning of the century by the British 
Statistician Karl Pearson, and, secondly, 
Spearman’s correlation coefficient (rho). The 
first is more powerful, but the data observed 
must be normally distributed and _ their 
variances must be homogeneous. Furthermore, 
the calculation takes longer unless a computer 
is available. The second, whose formula is: 


6s 4 
Tno==—] NIN@SS 196 1 
(D = difference between ranks; N = number 
of pairs) is less powerful but does not assume 
any preliminary condition of the data and is 
easier to calculate. Unless otherwise specified, 
“correlation” usually means “coefficient of 
correlation”, which ranges from 0 (showing 
complete absence of relationship) to 1 (de- 
noting perfect correspondence). It may be 
either positive or negative so that it ranges 
from -1 to +1. It is recommended that a 
Statistics manual or a statistician be con- 
sulted before ‘using these formulae. See 
Correlation. 
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Coefficient of reliability 


The coefficient of correlation between two 
forms of test, between scores on repeated 
administration of the same test,,or between 
halves of a test, properly corrected. These 
three coefficients measure somewhat different 
aspects of reliability. See, for example, the 
Kuder-Richardson formula. 234 


Competence 


The professional ability required to carry out 
certain functions. Recognised aptitude to 
perform a specific act. Competence is a 
potential which is realised at the moment of 
perronmancte = 11. 15 9-1 tl te 
134, 218; 236,.246, 302, 310, 319, 334233), 
343, 361-363, 409, 411-412, 441, 443, 468, 
480. 


Completion item 


Test question requiring completion of an 
unfinished sentence or phrase. 428 


Concept 

A general and abstract mental representation 
of something; a conceptual model. 102, 
202 200; 92.) 304s GUG,5 JU. 5 eo seek 
329; Si/¢ DIY tIO1., 505, 200, 309, 319 tag. 
428, 435, 478, 507. 


Continuous evaluation 215-216, 449-452 


Contract. See Programme/contract 

Contrast error 453 

Coordinator, qualities of | 370 

Contributing objective, see Enabling objective. 


Correlation 


Relationship or ‘“‘going-togetherness’’, between 
two scores or measures. The existence of a 
high correlation between two variables does 
not necessarily indicate that one has any 
causal influence on the other. See also Co- 
efficient of correlation. 467 


Criterion 


A standard by which a test may be judged or 
evaluated; a set of scores, readings, etc., that 
a test is designed to predict or to correlate 
with. See Validity. 12-13, 102, 109, 122. 


129, 143-144, 149-152, 155-156, 157, 160, 
204, 212-213, 221, 234, 238, 334, 336, 
SOV ISO2=303, 979, 3012 "402)°40F7'410, 
422, 424-425, 461-463, 472, 479-480, 503, 
507 


Curriculum 


Plan worked out in advance fixing the order or 
the timetable of a group of educational ac- 


tivities. 310, 313,314 
Defects (Limitations and outside factors 


affecting tests and examinations) 228-231 


Demonstrations 342 (a) 


Diagnostic evaluation See Formative evalu- 
ation 


Diagnostic test 


In education, a test used to ‘“diagnose’’, that 
is to locate, specific areas of weakness or 
strength and to determine the nature of the 
weaknesses or deficiencies; it yields measures 
of the components or sub-parts of some larger 
body of information or skill. Diagnostic 
achievement tests are most commonly pre- 
pared for assessing skills. See Formative 
evaluation 


Diascope See Overhead projector. 345 


Difficulty index 


The percentage of a specified group of 
students that answers an item correctly. Also 
called Facility index. 13, 15, 402, 461, 
465, 468-471, 476, 480 


Discrimination index 

The discriminating power of an item. 6, 
13, 15, 236, 247, 402, 461, 465, 468-471, 
475-476, 480, 503 


Distractor 

Any of the incorrect choices in a multiple- 
choice question. 228, 431, 440, 462, 
477 


Distribution (Frequency distribution) A 
tabulation of scores from high to low or low 


to high, showing the number of individuals 
that obtain each score or fill in each score 
interval 350-352, 416, 453, 467 


E 


Education 


Action or process of encouraging the form- 
ation and development of a person’s physical, © 


intellectual and moral faculties. 4, 7, 13, 
104-108, 111, 113-116, 123-125, 137, 


205, 211, 212, 215, 233, 302, 305-312, 315, 
318-319, 341, 363, 369-371, 410, 504, 
213 


Educational 


Directed towards education. Used to qualify 
the objectives defining student behaviour. 
8, 12-13, 102, 104-108, 117-120, 123-125, 
129, 132-133, 135, 137, 143-144, 148, 149- 
£50, 152-160; 163,202, 205721622 5eee. 
£38, 243, 305-313). 3155 3400 g0 ae 
360, 361-364, 369, 377-383, 402-404, 
410-411, 419, 424-425, 502, 513-514 


Effectiveness 


Capacity to produce the desired result. See 
Efficiency. 104, 302, 306-307, 310-312, 
408-409, 415, 461, 502, 506, 510 


Efficiency 

Capacity to produce the desired result at least 
cost. A system that is efficient is preferable 
to one that is merely effective as there will be 
no waste of resources. 233, 234, 237, 
336, 341, 358-359, 382 


Enabling objective 134, 143 


Epidiascope 344, 380 
Equilibrium (Of an examination). 234, 
230 


Equivalent form 


Any of two or more forms of a test that are 
closely parallel with respect to the nature of 
the content, the difficulty of the items 
included and their discriminating power, 
and that will yield very similar average scores 
and measures of variability for a given group. 
Used for studies on reliability of a test. 450 


Error of measurement See Standard error. 
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Essay question. See also Modified essay 
question 428-429, 475 


Evaluation 

Provides the basis for value judgments that 
make better educational decision-making 
possible. Includes a “measurement” _com- 
ponent and a “judgment and_ decision’”’ 
component. 12-13, 104; 106-109, 125, 
129, 133,9137-139,; 144; 202-209, 211-222, 
233, 238-243, 245-246, 307, BlO=3A72, 
315, 317, 342, 348, 353, 370-371, 402-406, 
409-410, 413-418, 421, 423, 434, 441, 443, 
450, 452-453, 461-462, 469, 506, 511, 
54025-0290, 0295951 


Explicit 
Stated in a sufficiently clear and precise 
manner, so as to leave no doubt. ANT: 


implicit, confused, equivocal, ambiguous, 
obscure. 111,139; 150, 163, 246, 334, 
353 

Factor 


In mental measurement, a hypothetical trait, 
ability, or component of ability that under- 
lies and influences performance on two or 
more tests, and hence causes scores on the 
test=-to be “correlated ‘The terimi='“factor’’ 
strictly refers to a theoretical variable, derived 
by a process of factor analysis from a table 
of interrelationships among tests; but it is 
also commonly used to denote the psycho- 
logical interpretation given to the variable 
hee ine mental! trait assimed to be 
represented by the variable, as verbal ability, 
numerical ability, etc. 133, 250; 234, 
336, 351, 368, 411, 453, 475 


Factor analysis 


A group of statistical methods for analysing 
the intercorrelations among a set of variables 
(for example, test scores). Factor analysis 
uncovers factors which explain the common 
characteristics of and the differences between 
the examinees and the questions. Factor 
analysis has contributed to the understanding 
of the organisation or components of intelli- 
gence, aptitudes and personality, and it has 
pointed the way to the development of tests 


6.06 


of the several components. ‘‘Factor’’ has no 
causal meaning here, it is used in its mathe- 
matical sense and is not intended to express 
a causal relationship. 475 


Feedback 


An informative reaction to the message of 
teaching. It is useful not only as a means of 
checking but also as a means of increasing 
the student’s learning activity. 216-219, 
327, 450-451, 529 


Field work 342a 


Final testing 449-451 
Flannelboard (Flannelgraph) 344 


Follow-up evaluation See Formative evalu- 
ation. 


Follow-up test See Post-test 


Formative evaluation 12> 108 9202 21 
24 i219 23193) 3, 9022504, as 


Function, professional 


Set of activities (directed towards the same 
aim) that a person performs to fulfil his role 


in society. 6, .42}4 5omO2esdt0 5 ,cr0Sjaidd, 
11260c121-127;:0129, 134-135; @s9u507, 
331, 333-337, 339, 348-349, 361, 363- 


364, 367, 377, 385, 404, 405, 411 


G 


Goal See Aim 


Groups, high and low 


H 


Halo effect 


468-469, 471, 476 


An interference effect created by the idea the 
examiner has already formed of a student on 
the basis of previous test results. 453 


Incident analysis 109, 441 


Indicator, health 


Indirect measure of the level of health of the 
community. 


Institutional objectives See Objectives general 


Instruction 


Process of communicating knowledge and 


training the mind of aperson 113, 346 
Integrated learning 355, 361-365 
Integrated teaching 355, 359-360 


Integration 


To join parts together to form a consistent 
whole. Coordination of different learning/ 
teaching activities with a view to the 
harmonious functioning of the educational 
process and more effective training of per- 
sonnel. See Integrated teaching and Integrated 
learning. 307, 310-311, 313, 408 


Intellectual process 


Process of thinking in general ideas or 
concepts; process of using knowledge, inter- 


preting data, solving problems, etc. 136- 
140, 227, 402, 435 

Interpretation of data 138-139, 227, 
435 

Interval testing 450-451 

Item 

A single question or exercise inatest. 236, 
465 


Item analysis See Question analysis. 


K 


Knowledge Seare. 102." 10a. DUy, 1 ht, 
133, 135-138, 143, 155, 157, 160, 212, 229, 
231, 306-309, 318, 321, 335, 351, 362, 370, 
379, 427, 432, 450, 527, 528 


Kuder-Richardson formula 


Formula for estimating the reliability of a test 
from information about the individual items 
in the test, or from the mean score, standard 


deviation, and number of items in the test. 
Because the Kuder-Richardson formula per- 
mits estimation of reliability from a single 
administration of a test, without it being 
necessary to divide the test into halves, its 
use has become common in test development. 
It is not appropriate for estimating the 
reliability of speed tests. The Kuder-Richard- 
son formula (No. 21) is as follows: 


wt. _k_ { 1-M(k-M) 
k-1 kS? 


k = number of questions in test 


= arithmetic average of scores obtained by 
the student 


S = standard deviation of scores. 


Law of effect, Thorndike’s 214, 245 
Learning 136)-107 Ad 75h 36jol89 salsa 
158;. 203, 212, 214-216. 302.4 3063 1a 
313, 318, 324-334, 341-349, 353, 360-362, 
363, 365, 379-382, 409, 412, 416-417 


Lecture 


A lesson given orally by a teacher, with vir- 
tually no student participation. It could be, 
and often is, distributed in printed form. This 
lack of real participation is the main characte- 
ristic of the lecture zz its traditional 
form: , 160;.335, B45 bes Aebmane 


Leniency, errorsof 453 


Logical error 453 


MCQ See Multiple-choice question. 


Machine-scorable (machine-scored) test 


A test which may be scored by means of a 
machine. In taking tests that are to be scored 
by machine the examinee records his answers 
on separate answer sheets. The machine 
rapidly distinguishes between right and 
wrong answers and can combine groups of 
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responses in order to yield total or part 
scores or carry out an item analysis with 
difficulty and discrimination indices. — 442 


Matching item 

Type of multiple-choice item calling for the 
correct association of data given in one list 
with those given in a second list. There are 
several types: simple matching, excluded 
term matching, causal relationship, quantit- 


ative comparison, relative variation. 432- 
433 

Media, teaching 341-347 

Modified essay question 1 2—de3 ced, 202) 


227, 402, 421, 428, 429-430, 450 


Motivation 


The conscious and unconscious forces that 
determine behaviour (independently of any 
moral consideration). 135, 214, 218, 
307,309, 311, 327, 348, 379; 502 


Multiple-choice question (MCQ) 


A test item in which the examinee’s task is 
to choose the correct or best answer from 
several given answers or options (distractors). 
12-13, 15, 202, 227-228, 402, 421, 428, 
431-434, 441-443, 450, 462-463, 477 


Multiple response items 


Type of multiple choice question in which 
two or more of the given choices may be 
correct and where there is generally a code 
permitting only one answer to be given on the 
answer sheet. — 432 


N 


N — The symbol commonly used to represent 
the number of cases in a distribution, study, 
etc. 241, 349, 468 


Non-directive method 


Teaching method whereby the teacher gives 
no direction but adopts an attitude which 
encourages students asking for advice to 
work out solutions to their problems them- 
selves. 318 
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Norm-referenced tests See Relative criteria 


tests. 


O 


Objective, educational 8, 12-13, 15, 101-102, 
104, 106-108, 110-111, 117-122, 123-125, 
129) 12-133. jihsose Life, 143-144, 
148-150, 152-161, 163, 202, 204, 207-208, 
2142 245=216) 220, 223, 225-220, 22, 
238, 243, 245, 302, 305-308, 311, 315, 317, 
334, 341-342, 348-349, 357, 359, 362-364, 
367-369, 373, 380-383, 402-404, 410-414, 
421, 422, 450, 461, 463-464, 514 


Objective, learning 117, 157, 333-334 


Objective, “specific (= '71252102)5 11828193 
129, 135, 143-144, 149-150, 152, 154-156, 
160, 163, 202, 204, 241, 348-349, 357- 
358, 369, 416, 503 


Objective, teaching 117 


Objective test 


A test in the scoring of which there is no 
possibility of differences of opinion among 
scorers as to whether responses are to be 
scored right or wrong. It is contrasted with 
tests such as the traditional essay examination 
to which different scorers may assign 
different scores, ratings or grades. Examples: 
MCQ, programmed examination, restricted 
response test. See these terms. 


Objectives, general 12, 102, 123, 157, 241 


Objectives, institutional See Objectives, 
general. 
Objectives, intermediate 118-119, 124- 


125, 129, 241, 348, 366, 404 


Objectivity 12, 163, 202, 230-231, 233- 
238, 246-247, 452, 503 


Observational rating scale See Scale. 
Obstacles to change 302, 375. 


Oral examination 222, 427 


Organisational diagram 


A diagram or chart showing the structure of 
a complex organisation, representing its 
various components and the relationships 
between them. The organisational diagram 
shows the place and responsibilities of each 
department or section. 12, 108, 202, 241, 
374 


Overhead projector 


An apparatus for projecting images from 
transparencies onto a screen so that they 
are visible in ordinary daylight. Also called 
Diascope. 345, 347, 516 


p 


Peer learning 


The process of students assisting in the teach- 
ing /learning system; an educational method 
which involves peers actively in the planning, 
implementation and evaluation of the educa- 
tional process. 135, 319, 337, 355, 363, 415 


Performance 


Accomplishment of an act (task) by a person. 
It is not used in the Handbook in the sense 
of establishing a record. Individual result 
obtained in carrying out a task, depending 
largely on aptitude and motivation. Level of 
performance = degree of individual success. 
13, 108, 110, 134, 143-144, 150, 154, 157, 
20A-T DIG, t22dynie 25i, 230; 2335.246, 33,3, 
337, 363-364, 402, 404, 409, 421, 425, 
429-430, 461-464, 469, 480, 503 


Post-test (follow up test) 449-452, 511, 529 


Practicability (Of a test) 225, 233-235, 


440 


Practical tests 421-422 


Practical work 342a 


Practice effect 


The influence of previous experience with a 
test on a later administration of the same 
test or a similar test. Usually there is an 
increase in the score on the second testing, 
attributed to increased familiarity with the 
directions, kinds of question, etc. Practice 
effect is greater when the interval between 


testings is short, when the materials in the 
two tests are very similar, and when the 
initial test is a relatively novel experience 
for the subjects. 416 


Pre-final feedback comprehensive _ testing 
449-451 


Prerequisite level test 


In education, a test that measures the extent 
to which an individual has acquired a certain 
level of competence needed to undertake 
some new learning activity successfully. For 
example, a prerequisite level test in pharm- 
cology shows the extent to which a student 
has reached a level whereby he may profitably 
begin a course on therapeutics. Also called 
readiness test. 144 


Pre-test 


Test which shows to what extent a student 
has acquired certain competences (know- 
ledge and skills) expected from a course 
before the commencement of the corres- 
ponding teaching. 9-10, 449-451 


Problem-solving 13,..130-131, 137-139; 
214, 227, 247, 306-311, 319, 335, 382, 
402, 412, 436, 439, 441, 478 


Programme 


A series of planned educational activities a stu- 
dent is to go through with the assistance of 
teachers. 13, 104, 106, 108, 111, 117, 133, 
204, 208, 213,.302-503; 306, 313s 17a o2ts uns 
368-369, 371, 373-374, 382-383, 385, 402, 
405-409, 413 


Programme/contract 363 


Programme preparation (For educational 
workshop). 519-522 


Programmed examination 13,,.15,. 402, 
421, 441-448, 478 


Programmed teaching 


Method of instruction consisting of a system 
of graduated questions with an immediate 
check of the answer; the right answer is 
needed in order to answer the following 
question correctly. Such teaching can be 
automated by using, for example, teaching 
machines or computers. 341, 343, 383, 
441-448 
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Project 222, 421-422 

Prospective (Studies) — concerning the future; 
research on the future evolution of mankind, 
predictions on future conditions — 110, 


133, 315 
Proximity error 453 


Psychomotor See Skill 


Q 


Question (or item) analysis 

The process: of evaluating a single test item 
by any of several methods, in particular by 
determining its difficulty index, discrimin- 
ation index and often its correlation with 
some selected criteria. 13, 402, 465- 
473 


Question analysis card 473 
Question bank 473 


Questionnaire (For evaluation of workshop) 
415, 525-531 


R 


Random sample 


A sample of the members of a population 
drawn in such a way that every member of 
the population has an equal chance of being 
included. This method precludes any bias or 
selection. The purpose of using the method 
is, of course, to obtain a fairly “representative”’ 
sample of the total population, so that the 
sample findings may be generalised to the 
whole population. Random sampling has 
also the following advantage: in accordance 
with the calculus of probabilities, formulae 
are available for predicting the relative fre- 
quencies or parameters, such as the mean or 
standard deviation of the sample from the 
true values of the relative frequencies or 
parameters in the total population; con- 
versely, the limits within which are very 
probably situated the relative frequencies 
or parameters of the population from which 
the sample is drawn can be estimated from 
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the, relative frequencies or parameters 

observed within the sample. There are various 

methods, in particular random number 

tables, for drawing such a sample in practice. 
349-350 


Range 

The difference between lowest and highest 
scores obtained in a test administered to a 
given group. 467 


Rank-ordering 


The process of arranging students in order of 
merit as determined by scores obtained in 
one or more tests. 245, 265, 467-468, 
471-472 7 


Raw score . 


The first quantitative result obtained in 

scoring a test (for example, the number of 

right answers, number of errors, or similar, 

direct, unconverted, uninterpreted measures). 
465-467 


Recall item 


An item that requires the examinee to supply 
the correct answer from memory. This item 
differs from the recognition item in which 
he need only identify the correct answer. 
For example, “Claude Bernard published 
the ‘Introduction to the Study of Experi- 
meéntal Medicine” im" 4 0% >. PPh Rig ee TCCAL 
item, whereas ‘Claude Bernard published 
the ‘Introduction to the Study of Experi- 
mental Medicine’ in (a) 1785, (b) 1815, (c) 
1865, (d) 1905” is a recognition item in 
the form of a multiple-choice question. 13, 
402, 428, 435 


Relative criteria test 
479 


13, 461-465, 472, 


Relevance 


Quality of being perfectly appropriate to the 
object pursued. In the context of the Hand- 
book, the extent to which training pro- 
grammes correspond to the health problems 
of the community and the resources available. 

12-13, 15, 102, 111, 129, 133, 143, 148, 
193500163, ..202, 4225, 234, 2367237,,.239; 
247° 302 5.93935 358}, 3095, 873, 885,428, 
452,526 


Reliability 


The extent to which a test is consistent in 
measuring whatever it has to measure; 
dependability, stability, and relative freedom 
from errors of measurement. Reliability is 
usually estimated by some form of correlation 
coefficient (see Coefficient of reliability , 
Standard error). 12,6202, 231, 233-237. 
247, 452, 475, 478 


Representative sample 


A sample that corresponds to or matches the 
population of which it is a sample with 
respect to characteristics important for the 
purposes ofthe investigation. 110 


Restricted response test See Modified essay 
question. 


Role, professional 


The functions fulfilled by aperson. 134 


Role, teacher’s 13, 333-337, 341 


Role playing 
Group learning/teaching technique in which 
students learn to approach a problem situation 


by acting out freely the roles of the pro- 
tagonists in the situation. 342a 


S 


Safety evaluation 451 


Scale 


Continuous series of programmed values in 
which data are situated in order to assess 
them with respect to a whole. Also called 


Observational rating scale. 143, 222, 415- 
418, 421-425, 452-453 

Score See Raw score. 

Scoring, alphabetical or numerical 224, 


231, 428, 440, 442, 464-467 
Selection (Of students) 423 


Self-learning package 13, 302, 343, 348- 


354, 380 


Seminars 341 


Sensorimotor See Skill. 


Short open answer question, See Modified 
essay question 


Simulation 222, 343, 440 


Skill, professional 


Can be an intellectual skill (cognitive domain), 
a communication skill (affective domain) or 
a practical skill (sensorimotor or psychomotor 
domain, referring simultaneously to  sen- 
sitivity, sensation and motricity). 12-13; 
102, 109-111, 134-141, 143-144,154-156, 
158, 163, 221-222, 230, 9305-312. 397. 
324— 325, 353, 361-366, 402, 408-409, 
412, 422-423, 459 


Slides 345, 348, 380 


Spearman-Brown formula 


A formula giving the relationship between 
the reliability of a test and its length. 


This formula 


Zt 
(te 
eee oem | 
r'’ = coefficient of reliability 
r = coefficient of correlation between 


the two halves of the test 


permits estimation of the reliability of a test 
lengthened or shortened by any amount, from 
the known reliability of a test of specified 
length. Its most common application is in 
the estimation of reliability of an entire test 
from the correlation between the two halves 
of the test (split-half reliability). 

Specification table P30 Vi27, 250, waa 
358, 373, 454-457 


Specificity (Of atest) 234, 236-237 


Stages (Of assessment) 13, 402, 449-452 


Standard error 


The standard error is the standard deviation 
of a sampling distribution. The use of the 
standard error is very important when the 
sampling distribution follows the normal 
distribution law (i.e. when the sample is 
drawn at random and is sufficiently large: 
for example, n > 30 in the case of the 
sampling distribution of a mean). 


Its formula is as follows: 


E = S<VJ1-rt 
E =~ standard error 
S = standard deviation 


coefficient of reliability 
See Kuder-Richardson formula. 


Summative evaluation See Certifying evalu- 
ation. 


Supervision 

Guidance and monitoring of activities of 
health personnel to ensure an acceptable 
level of performance. 108, 134 


System 


A coherent group of interacting abstractions 
and components forming both a theoretical 
structure and practical method. Sub-system: 
partofasystem. 108 


Systematic approach 


Theory highlighting the interdependence of 

the components of something seen as a whole; 

approaching the elements of a problem by 

considering them as an interdependent whole 
a7 02-503, 528 


Systems analysis 


Process of studying an activity by mathe- 
matical means in order to define its goals and 
purposes and to discover ways of accom- 
plishing it more efficiently; a constructive 
method in three phases: collection of data, 
analysis, and synthesis leading to solving 
of problems. 110, 233, 371 


T 


Task, professional. 


A specific piece of work that has to be done. 
6, 8, 12, 102, 105-108, 110-111, 117-120, 
134-136, 141, 143, 149-150, 152, 154-156, 
158, 160, 227, 240, 306-307, 314-317, 
399, 9I9, 301-364, 366,’ 385) 405-406; 
408-409, 425-426, 450, 461, 480, 510 


Task analysis 109-111, 307 
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Taxonomy 


Classification into categories according to a 
given system. See Classification. 


Teaching 


Helping a person to know how to act, think 
or feel. 6, 12-13, 105, 107-108, 117, 
125, 127, 133, 136-137, 139, 144, 146, 157, 
163; *202:° 212, 219, °246;"S02Z; "306, 313, 
315, 317; 318-323, S834337,94349947; 
357-360, 362-363, 367-372, 381-383, 385, 
402, 408-413, 415, 441, 502-503, 533 


Teaching methods 


All the means applied in a rational manner to 
produce a specified educational result. 13, 
108, 133, 139, 157, 228, 302, 341, 340-347; 


353,.°355; .357-358, 369, 373, suicgae, 
533 
Techniques, teaching 341-354, 415 


Test 


Exercise, examination or work on the basis 
of which a judgment can be made as to a 
student’s intellectual, practical.or communi- 
cation skills. 9, 12-13, 202, 225-231, 
233, 240-241, 247, 311, 363-364, 373, 
402, 421-422, 425-426, 434, 441-444, 
449-452, 461-465, 511, 529 


Test(s) (Advantages and _ limitations of 
different types of test). See Defects. 


Time factor 133 


Transparency 


Sheet of transparent cellulose or acetate film 
(thermosensitive or otherwise) which can be 
written or drawn upon. The image can then 
be projected onto a screen by an overhead 
projector without darkening the room.— 345, 
347, 516 te 


Tutorials 341 


V 


Validity 233, 235-239 247 


9 


W 


Working methods (In educational workshop) 
7, 331, 309-521, 522; 524; 526, 530 


Workshop (Definition, aims) G07, S407, 
152, 333, 341, 416, 302-533 
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WHO publications may be obtained, direc 
ALGERIA: Entreprise nationale du Livre (ENAL), 3 bd Zirout Youcef, 
ALGIERS 


ARGENTINA: Carlos Hirsch SRL, Florida 165, Galerias Giiemes, Escri- 
- torio 453/465, BUENOS AIRES 


RALIA: Hunter Publications, 58A Gipps Street, COLLINGWOOD, 
Wc 3066 — Australian Government Publishing Service (Mail order 
sales), P.O. Box 84, CANBERRA A.C.T. 2601; or over the counter from 
Australian Government Publishing Service Bookshops at: 70 Alinga 
Street, CANBERRA CITY A.C.T. 2600; 294 Adelaide Street, BRISBANE, 
Queensland 4000; 347 Swanston Street, MELBOURNE, VIC 3000; 
309 Pitt Street, SYDNEY, N.S.W. 2000; Mt Newman House, 
200 St. George’s Terrace, PERTH, WA 6000; Industry House, 12 Pirie 
Street, ADELAIDE, SA 5000; 156-162 Macquarie Street, HOBART, 
TAS 7000 — R. Hill & Son Ltd, 608 St. Kilda Road, MELBOURNE, 
VIC 3004; Lawson House, 10-12 Clark Street, CROW’S NEST, 


NSW 2065 
AUSTRIA: Gerold & Co., Graben 31, 1011 VIENNA I 


BANGLADESH: The WHO Representative, G.P.O. Box 250, 
DHAKA 5 


BELGIUM: For books : Office International de Librairie s.a., avenue Mar- 
nix 30, 1050 BRUSSELS. For periodicals and subscriptions : Office Inter- 
national des Périodiques, avenue Louise 485, 1050 BRUSSELS — Sub- 
scriptions to World Health only: Jean de Lannoy, 202 avenue du Roi, 
1060 BRUSSELS 


BHUTAN: see India, WHO Regional Office 
BOTSWANA: Botsalo Books (Pty) Ltd., P.O. Box 1532, GABORONE 


BRAZIL: Centro Latinoamericano de Informagd4o em Ciencias de Saude 
‘(BIREME), Organizaga4o Panamericana de Saude, Sector de Publicagées, 
‘C.P. 20381 - Rua Botucatu 862, 04023 SAO PAULO, SP 


BURMA: see India, WHO Regional Office 


CANADA: Canadian Public Health Association, 1335 Carling Avenue, 
Suite 210, OTTAWA, Ont. K1Z 8N8. (Tel: (613) 725-3769. 
Télex : 21-053-3841) 


CHINA: China National Publications Import & Export Corporation, P.O. 
Box 88, BELJING (PEKING) 


DEMOCRATIC PEOPLE’S REPUBLIC OF KOREA: see India, WHO 
Regional Office 


DENMARK: Munksgaard Export and Subscription Service, Nerre 
Sogade 35, 1370 COPENHAGEN K (Tel: +45 1 12 85 70) 


FIJI: The WHO Representative, P.O. Box 113, SUVA 


FINLAND: Akateeminen Kirjakauppa, Keskuskatu 2, 
HELSINKI 10 


FRANCE: Librairie Arnette, 2 rue Casimir-Delavigne, 75006 PARIS 


GERMAN DEMOCRATIC REPUBLIC: Buchhaus Leipzig, Post- 
fach 140, 701 LEIPZIG 


GERMANY, FEDERAL REPUBLIC OF : Govi-Verlag GmbH, Ginnhei- 
merstrasse 20, Postfach 5360, 6236 ESCHBORN — Buchhandlung Alex- 
ander Horn, Friedrichstrasse 39, Postfach 3340, 6200 WIESBADEN 


GHANA: Fides Entreprises, P.O. Box 1628, ACCRA 


GREECE: G.C. Eleftheroudakis S.A., Librairie internationale, rue Ni- 
_ kis 4, ATHENS (T. 126) 


HONG KONG: Hong Kong Government Information Services, Beacons- 
field House, 6th Floor, Queen’s Road, Central, VICTORIA 


HUNGARY: Kultura, P.O.B. 149, BUDAPEST 62 


INDIA: WHO Regional Office for South-East Asia, World Health House, 
Indraprastha Estate, Mahatma Gandhi Road, NEW DELHI 110002 


INDONESIA: P.T. Kalman Media Pusaka, Pusat Perdagangan Senen, 
Block 1, 4th Floor, P.O. Box 3433/Jkt, JAKARTA 


IRAN (ISLAMIC REPUBLIC OF): Iran University Press, 85 Park 
Avenue, P.O. Box 54/551, TEHERAN 


IRELAND: TDC Publishers, 12 North Frederick Street, DUBLIN 1 
(Tel: 744835-749677) 


ISRAEL: Heiliger & Co., 3 Nathan Strauss Street, JERUSALEM 94227 
ITALY: Edizioni Minerva Medica, Corso Bramante 83-85, 10126 


Beoad Via Lamarmora 3, 20100 MILAN; Via Spallanzani 9, 00161 


00101 


ire Maruzen Co. Ltd, P.O. Box 5050, TOKYO International, 


JORDAN : Jordan Book Centre Co. Ltd., University Street, P.O. B ! 
(Al-Jubeiha), AMMAN : iis 


KUWAIT: The Kuwait Bookshops Co. Ltd, Thunayan Al-Gh 
P.O. Box 2942, KUWAIT anem Bldg, 


LAO PEOPLE’S DEMOCRATIC REPUBLIC: The WHO Programme 
Coordinator, P.O. Box 343, VIENTIANE 


t or through booksellers, from: 


LUXEMBOURG: Librairie du Centre, 49 bd Royal, LUXEMBOURG 


MALAWI: Malawi Book Service, P.O. Box 30044, Chichiti, 
BLANTYRE 3 


MALAYSIA: The WHO Representative, Room 1004, 10th Floor, Wisma 
Lim Foo Yong (formerly Fitzpatrick’s Building), Jalan Raja Chulan, 
KUALA LUMPUR 05-10; P.O. Box 2550, KUALA LUMPUR 01-02 — 
Parry’s Book Center, 124-1 Jalan Tun Sambanthan, P.O. Box 10960, 
KUALA LUMPUR 


MALDIVES: see India, WHO Regional Office 


MEXICO: Libreria Internacional, S.A. de C.V., av. Sonora 206, 06100- 
MEXICO, D.F. 


MONGOLIA : see India, WHO Regional Office 
MOROCCO: Editions La Porte, 281 avenue Mohammed V, RABAT 
NEPAL: see India, WHO Regional Office 


NETHERLANDS: Medical Books Europe BV, Noorderwal 38, 7241 BL 
LOCHEM 


NEW ZEALAND: New Zealand Government Printing Office, Publishing 
Administration, Private Bag, WELLINGTON ; Walter Street, WELLING- 
TON; World Trade Building, Cubacade, Cuba Street, WELLINGTON. 
Government Bookshops at : Hannaford Burton Building, Rutland Street, 
Private Bag, AUCKLAND; 159 Hereford Street, Private Bag, CHRIST- 
CHURCH; Alexandra Street, P.O. Box 857, HAMILTON; T & G Build- 
ing, Princes Street, P.O. Box 1104, DUNEDIN — R. Hill & Son, Ltd, 
Ideal House, Cnr Gillies Avenue & Eden St., Newmarket, AUCK- 
LAND 1 


NORWAY: Tanum — Karl Johan A.S., P.O. Box 1177, Sentrum, N-0107 
OSLO 1 


PAKISTAN: Mirza Book Agency, 65 Shahrah-E-Quaid—E~Azam, P.O. 
Box 729, LAHORE 3 


PAPUA NEW GUINEA: The WHO Representative, P.O. Box 646, 
KONEDOBU 


PHILIPPINES : World Health Organization, Regional Office for the Wes- 
tern Pacific, P.O. Box 2932, MANILA 


PORTUGAL: Livraria Rodrigues, 186 Rua da Ouro, LISBON 2 


REPUBLIC OF KOREA: The WHO Representative, Central P.O. 
Box 540, SEOUL 


SINGAPORE: The WHO Representative, 144 Moulmein Road, SINGA- 
PORE 1130; Newton P.O. Box 31, SINGAPORE 9122 


SOUTH AFRICA: Contact major book stores 


SPAIN: Ministerio de Sanidad y Consumo, Centro de Publicaciones, 
Documentacién y Biblioteca, Paseo del Prado 18, 28014 MADRID — 
Comercial Atheneum S.A., Consejo de Ciento 130—136, 08015 BARCEL- 
ONA; General Moscardé 29, MADRID 20 — Libreria Diaz de Santos, 
P.O. Box 6050, 28006 MADRID; Balmes 417 y 419, 08022 BARCEL- 
ONA 


SRI LANKA: see India, WHO Regional Office 


SWEDEN: For books: Aktiebolaget C.E. Fritzes Kungl. Hovbokhandel, 
Regeringsgatan 12, 103 27 STOCKHOLM. For periodicals : Wennergren- 
Williams AB, Box 30004, 104 25 STOCKHOLM 


SWITZERLAND: Medizinischer Verlag Hans Huber, Langgassstrasse 76, 
3012 BERN 9 


THAILAND: see India, WHO Regional Office 


UNITED KINGDOM: H.M. Stationery Office: 49 High Holborn, LON- 
DON WCIV 6HB;; 13a Castle Street, EDINBURGH EH2 3AR;; 80 Chi- 
chester Street, BELFAST BT1 4JY ; Brazennose Street, MANCHESTER 
M60 8AS;; 258 Broad Street, BIRMINGHAM B1 2HE; Southey House, 
Wine Street, BRISTOL BS1 2BQ. All mail orders should be sent to: 
HMSO Publications Centre, 51 Nine Elms Lane, LONDON SW8 
5DR 


UNITED STATES OF AMERICA : Copies of individual publications (not 
subscriptions): WHO Publications Center USA, 49 Sheridan Avenue, 
ALBANY, NY 12210. Subscription orders and correspondence concerning 
subscriptions should be addressed to the World Health Organization, 
Distribution and Sales, 1211 GENEVA 27, Switzerland. Publications are 
also available from the United Nations Bookshop, NEW YORK, 
NY 10017 (retail only) 


USSR : For readers in the USSR requiring Russian editions : Komsomolskij 
prospekt 18, Medicinskaja Kniga, MOSCOW — For readers outside the 
USSR requiring Russian editions : Kuzneckij most 18, MeZdunarodnaja 
Kniga, MOSCOW G-200 


VENEZUELA : Libreria Médica Paris, Apartado 60.681, CARACAS 106 


YUGOSLAVIA : 27/Ml, 11000 


Jugoslovenska Knijiga, 
BELGRADE 


Terazije 


Special terms for developing countries are obtainable on applicati 
@ ob pplication to the WHO Representatives or WHO Regional Offices 
listed above or to the World Health Organization, Distribution and Sales Service, 1211 Geneva 27, Switzerland, Orders from 


countries where sales agents have not yet been appointed ma 
y also be sent to the Geneva address, but must be paid for in 
pounds sterling, US dollars, or Swiss francs. Unesco book coupons may also be used. " 


Prices are subject to change without notice. 
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